Stage 2 failure building generator

[simonjwright]

A colleague is building gcc-14.1-darwin-r0 in a Github runner (macos-14 runs on an M1-based machine) and has problems with the gen_il-main built in stage 2.

The stage1 compiler is aarch64-apple-darwin 13.2.0, which successfully builds and runs:

2024-05-27T16:01:53.6688970Z mkdir -p ada/gen_il
2024-05-27T16:01:53.6836290Z cd ada/gen_il; gnatmake -q -g -gnata -gnat2012 -gnatw.g -gnatyg -gnatU -I/Users/runner/work/GNAT-FSF-builds/GNAT-FSF-builds/sbx/aarch64-darwin/gcc/src/gcc/ada gen_il-main
[...]
2024-05-27T16:01:57.9128640Z cd ada/gen_il; ./gen_il-main
2024-05-27T16:01:58.1111820Z install.texi:261: warning: @anchor should not appear on @item line

The stage2 compiler builds the same:

2024-05-27T16:14:20.4760580Z mkdir -p ada/gen_il
2024-05-27T16:14:20.4823240Z cd ada/gen_il; gnatmake -q -g -gnata -gnat2012 -gnatw.g -gnatyg -gnatU -I/Users/runner/work/GNAT-FSF-builds/GNAT-FSF-builds/sbx/aarch64-darwin/gcc/src/gcc/ada gen_il-main

but when it’s run, this happens:

2024-05-27T16:14:24.8200660Z cd ada/gen_il; ./gen_il-main
2024-05-27T16:14:24.8266370Z dyld[74097]: Symbol not found: ___builtin_nested_func_ptr_created
2024-05-27T16:14:24.8326660Z   Referenced from: <A3324F46-453C-314E-B26B-51C847B1E704> /Users/runner/work/GNAT-FSF-builds/GNAT-FSF-builds/sbx/aarch64-darwin/gcc/build/gcc/ada/gen_il/gen_il-main
2024-05-27T16:14:24.8329370Z   Expected in:     <B3E386AD-E6E3-3A23-B4B3-C06CA85CFE57> /Users/runner/work/GNAT-FSF-builds/GNAT-FSF-builds/sbx/aarch64-darwin/gcc/build/prev-gcc/libgcc_s.1.1.dylib
2024-05-27T16:14:24.8337720Z /bin/sh: line 1: 74097 Abort trap: 6           ./gen_il-main
2024-05-27T16:14:24.8340570Z make[3]: [ada/stamp-gen_il] Error 134 (ignored)

Could the Xcode/CLT version make a difference? The runner is running 14.4.1, and the default Xcode is 15.0.1.

[iains]

I recall earlier Xcode 15 having quite a few problems (mostly with the new linker) that we worked around by configuring to use "ld-classic" .. If possible, update to Xcode 15.3 which is known to work (and has quite a few wrinkles ironed out). If that is not possible, then perhaps we can find a configure recipe that will use the 'classic' linker.

[simonjwright]

Our problem with 15.3 (well, CLT 15.3) is that it doesn’t provide m4 (it does provide gm4, though, so you can configure GMP with M4=gm4). I just ran the job locally with XC 15.4 (no CLT yet, though) and it succeeded.

Since this compiler is being provided to users, would it make sense to go for as early an XC version as actually builds the compiler? I’m trying XC 15.1 on Github as I write ...

[iains]

If you are trying to have one build that runs using several different XC installs, we are going to need to be careful - the compiler configuration determines the capabilities of the support toolchain (as, ld, dsymutil) and adjusts accordingly. Similarly, the SDK in use is relevant (since some of the SDKs require special handling).

If you are building the compiler, and providing it as a built item - then perhaps building m4 is not the end of the world.

For the record, I build GMP et. al as "in-tree" sources - they are bootrapped along with the compiler and I have not run into trouble with XC15.1b. As noted CLT 15.0 did have (dyld-linker) problems that are show-stoppers .. so .. you need to stop and take pause about what you want to offer and how to communicate that to your users.

edit: note that generally speaking toolchains are considered to consist of all the items in use - compiler, linker, assembler, debug linker. Xcode does not try to marry an arbitrary clang version with any arbitrary linker .. so perhaps you can constrain what you offer without being seen as providing a poor solution?

[simonjwright]

We’re building the compiler, and providing it to our users as a built item.

I build gmp etc in-tree, my colleagues out-of-tree, for reasons. Anyway, I ran export M4=gm4 and all worked for their build (of gmp, since we still have compiler issues).

I ran on Github with XC 15.1 and 15.3 - no change.

I noticed that using their successful x86_64 build of GCC 14.1 to compile this trampoline-using program

with Ada.Text_IO;
procedure Trampo is
   type T is access procedure;
   procedure P (The_T : T) is
   begin
      The_T.all;
   end P;
   procedure A is
   begin
      Ada.Text_IO.Put_Line ("foo.");
   end A;
begin
   P (A'Access);
end Trampo;

generated a reference to libgcc_s.1.1.dylib

$ otool -L trampo
trampo:
	@rpath/libgcc_s.1.1.dylib (compatibility version 1.0.0, current version 1.1.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.100.2)

in spite of the fact that it didn’t need to, because of linking with -lheapt_w.

My build of GCC 14.1.0 (using the FSF release) didn’t.

I think this might be down to a specs difference: theirs has

*libgcc:
%{static-libgcc|static:						      %:version-compare(!> 10.6 mmacosx-version-min= -lgcc_eh);		     shared-libgcc|fexceptions|fobjc-exceptions|fgnu-runtime:		     -lgcc_s.1.1						      %:version-compare(!> 10.3.9 mmacosx-version-min= -lgcc_eh)		      %:version-compare(>< 10.3.9 10.5 mmacosx-version-min= -lgcc_s.10.4)       %:version-compare(>< 10.5 10.6 mmacosx-version-min= -lgcc_s.10.5)	    } -lgcc

and mine has

*libgcc:
%{static-libgcc|static:						      %:version-compare(!> 10.6 mmacosx-version-min= -lgcc_eh);		     shared-libgcc|fexceptions|fobjc-exceptions|fgnu-runtime:		     %:version-compare(!> 10.11 mmacosx-version-min= -lgcc_s.1.1)						      %:version-compare(!> 10.3.9 mmacosx-version-min= -lgcc_eh)		      %:version-compare(>< 10.3.9 10.5 mmacosx-version-min= -lgcc_s.10.4)       %:version-compare(>< 10.5 10.6 mmacosx-version-min= -lgcc_s.10.5)	    } -lgcc

(the difference is that theirs has shared-libgcc|fexceptions|fobjc-exceptions|fgnu-runtime: -lgcc_s.1.1 whereas mine has shared-libgcc|fexceptions|fobjc-exceptions|fgnu-runtime: %:version-compare(!> 10.11 mmacosx-version-min= -lgcc_s.1.1))

Why that should result in their failure to build the aarch64 compiler I don’t know.

I found some interaction between gcc/configure and gcc/config/darwin.h to do with DARWIN_AT_RPATH, which looks as though it's related to the two different specs entries above.

[iains]

getting the specs right for the various permutations of Darwin's support for DYLD_LIBRARY_PATH has been somewhat of a labour.

Since (at least what I infer from your report) it seems that you are using different versions/branches/patches (?) it's very hard for me to figure out what might be wrong.

Really, the recommendation (for people who are not using homebrew, macports etc.) would be to use the branch(es) here - and if something does not work properly (or how you need it to) report issues so we can fix (or work around) them .. once we diverge significantly it's outside of what my meagre resources can handle :) and specs are a powerful tool ..

[simonjwright]

Both I and my colleagues are using gcc-14.1-darwin-r0, unpatched. The build script we are both using

• downloads a built 13.2.0 aarch64 compiler
• downloads gmp etc sources
• downloads gcc-14.1-darwin-r0
• builds gmp etc
• builds the 14.1 compiler

When run on Github, the compiler build crashes with the failure to run the gen_il-main built in stage 2, because of the problem I quoted at the top of this issue.

When run locally, the compiler build succeeds.

I’ve tried XC 15.1 locally and on Github, no change.

What I’m going to do next (after family commitments) is to compare the build logs.

After that, I’ll try to find why my own set of build scripts (again, using unpatched gcc-14.1-darwin-r0) doesn’t build gen_il-main using libgcc_s.1.1.dylib.

[iains]

Both I and my colleagues are using gcc-14.1-darwin-r0, unpatched.

That the specs are different is, in that case, pretty odd.

Having said that the configuration does adjust to the capabilities of the host system - including to whatever is detected in terms of linker capabilities***

The build script we are both using

• downloads a built 13.2.0 aarch64 compiler
• downloads gmp etc sources
• downloads gcc-14.1-darwin-r0
• builds gmp etc

On the GH/colleague's version this is a separate step - where you (and I) usually just build them in-tree? (not that I expect that to make the difference).

• builds the 14.1 compiler

When run on Github, the compiler build crashes with the failure to run the gen_il-main built in stage 2, because of the problem I quoted at the top of this issue.

Can we get the output of uname -a on the GH instance?
we might also need to get the versions of some key installed utilities - see *** below.

When run locally, the compiler build succeeds.

I’ve tried XC 15.1 locally and on Github, no change.

no change == builds locally, fails on GH?

What I’m going to do next (after family commitments) is to compare the build logs.

• configure logs would, perhaps be more revealing.

• also the difference between gcc/auto-host.h on the two systems.

After that, I’ll try to find why my own set of build scripts (again, using unpatched gcc-14.1-darwin-r0) doesn’t build gen_il-main using libgcc_s.1.1.dylib.

That could be quite an involved project - now I know you are starting from the same point - the thing is to figure out why the configure / build thinks that there's a difference.

===

*** some configure functionalities depend on the installed utilities (e.g. gawk, objdump, otool, etc.) further to that it can be that the linker used by the build compiler can be relevant.

Those are areas where there could be deviation between the environments.

I am understanding (hopefully) that the failing get_il-main is built by the stage #1 compiler (and not the host/bootstrap one)?

[iains]

if it is possible, please could you post the output of:

otool -lv gcc/ada/gen_il/gen_il-main |grep -A3 LC_RPA

from the failing case ..

.. IIUC, this is a $build tool (i.e. intended to run on the $build system which can be different from the $host one) and should be being built with the bootstrap/build-system compiler.

edit: there are several Ada build-time tools.

[iains]

please also post your configure lines (for both cases). I just took a look at my gcc-14.1 build and get_il-main seems to be correctly built with the bootstrap/build-system compiler at each stage.

[simonjwright]

I’m going to be away for a few days, but in the mean time

• for their build on Github,

../src/configure
--prefix=/Users/runner/work/GNAT-FSF-builds/GNAT-FSF-builds/sbx/aarch64-darwin/gcc/install
--enable-languages=c,ada,c++
--enable-libstdcxx
--enable-libstdcxx-threads
--enable-libada
--disable-nls
--without-libiconv-prefix
--disable-libstdcxx-pch
--enable-lto
--disable-multilib
--disable-libcilkrts
--without-build-config
--with-build-sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
'--with-specs=%{!sysroot=*:--sysroot=%:if-exists-else(/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk)}'
--with-mpfr=/Users/runner/work/GNAT-FSF-builds/GNAT-FSF-builds/sbx/aarch64-darwin/mpfr/install
--with-gmp=/Users/runner/work/GNAT-FSF-builds/GNAT-FSF-builds/sbx/aarch64-darwin/gmp/install
--with-mpc=/Users/runner/work/GNAT-FSF-builds/GNAT-FSF-builds/sbx/aarch64-darwin/mpc/install
--build=aarch64-apple-darwin23.5.0

• for my local build using my scripts,

/Volumes/Miscellaneous3/src/gcc-14-branch/configure
--prefix=/Volumes/Miscellaneous3/aarch64/gcc-14.1.0-aarch64
--without-libiconv-prefix
--disable-libmudflap
--disable-libstdcxx-pch
--disable-libsanitizer
--disable-libcc1
--disable-libcilkrts
--disable-multilib
--disable-nls
--enable-languages=c,c++,ada
--host=aarch64-apple-darwin21
--target=aarch64-apple-darwin21
--build=aarch64-apple-darwin21
--without-isl
--with-build-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
--with-sysroot=
--with-specs='%{!sysroot=*:--sysroot=%:if-exists-else(/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk)}'
--with-bugurl=https://github.com/simonjwright/building-gcc-macos-native
--enable-bootstrap
--enable-host-pie
CFLAGS=-Wno-deprecated-declarations
CXXFLAGS=-Wno-deprecated-declarations

• for the local build using their scripts,

$ otool -lv gcc/ada/gen_il/gen_il-main | grep -A3 LC_RPA
          cmd LC_RPATH
      cmdsize 32
         path @loader_path (offset 12)
Load command 17
          cmd LC_RPATH
      cmdsize 136
         path /Users/simon/Developer/GNAT-FSF-builds/sbx/aarch64-darwin/base_gcc/install/lib/gcc/aarch64-apple-darwin23.2.0/13.2.0 (offset 12)
Load command 18
          cmd LC_RPATH
      cmdsize 96
         path /Users/simon/Developer/GNAT-FSF-builds/sbx/aarch64-darwin/base_gcc/install/lib/gcc (offset 12)
Load command 19
          cmd LC_RPATH
      cmdsize 32
         path /opt/homebrew/lib (offset 12)
Load command 20
          cmd LC_RPATH
      cmdsize 96
         path /Users/simon/Developer/GNAT-FSF-builds/sbx/aarch64-darwin/base_gcc/install/lib (offset 12)
Load command 21
$

(the LC_RPATH refers to the host compiler).

• for my local build using my scripts,

$ otool -lv gcc/ada/gen_il/gen_il-main | grep -A3 LC_RPA
$

(i.o.w no LC_RPATH entries at all).

[simonjwright]

I am understanding (hopefully) that the failing get_il-main is built by the stage #1 compiler (and not the host/bootstrap one)?

Yes, it’s the one built in stage 2 by the compiler that was built by the host compiler in stage 1

[iains]

I am understanding (hopefully) that the failing get_il-main is built by the stage #1 compiler (and not the host/bootstrap one)?

Yes, it’s the one built in stage 2 by the compiler that was built by the host compiler in stage 1

but I don't think that is right .. it is an exe to run on the build system - it should be built with XXX_FOR_BUILD (which is effectively XXX_FOR_HOST when bootstrapping). I will have to see from your configure lines what is confusing the ada build into using the stageN compiler to build for $build (it does not happen for my configure lines, that exe is built with the bootstrap/build-system compiler in each stage)

[iains]

BTW : https://gcc.gnu.org/bugzilla/show_bug.cgi?id=79885 is the reason that I avoid --with-build-sysroot= ...

I see both configure lines have a bunch of (what I believe are) unnecessary configure flags - it is important to be very clear about why any non-standard flag is added - the configure script is supposed to get it right for the platform; if it is failing we should fix it and then remove any work-around.

[iains]

I would also suggest :
--build=aarch64-apple-darwin2? (edited)
for both cases

• $build will be copied to $host and then to $target if neither is provided.
• building for aarch64-apple-darwin23.5.0 is in conflict with the idea that exes should be ABI-compatible with any darwin23 version. That might be upsetting something.

I like your creative specs for the sysroot; but what would be better is an upstream BZ that says what the problem is so that, maybe, we can find a more efficient solution than checking for each invocation.

You are on different OS versions, it seems .. although TBH, I'd hope that does not affect things too much.

Are you using the singe additional fix posted to the gcc-14-1-darwin branch?
75ff8c3

that could, potentially fix issues with mishandling the SDK (although I'd expect a different kind of fail from it .. so maybe not relevant).

edit2: note also that gnatmake et. al. do not accept a --sysroot option .. so that has to be done at a lower level (hopefully, it is working OK).