Support Engine Images Again

[philip-klein]

Origin:

Running ue4-docker build --help suggests engine as a valid target still after #273

Possible Fixes:

• add the support for this target back (which I'm advocating for)
• remove engine from the help text

Before taking action, the two options should be weighed.

Weighing the options

Although there may not seem any use in the engine image, there is a functionality difference between an installed build and a source build of unreal.

For example:
My team requires a source build so that unreal can package individual plugins. This feature breaks with an installed build. Packaging a plugin allows me to send downstream developers packaged versions of my plugin instead of the source code. I use CI to verify this process on every merge in all of my team's plugins.

If I loose the ability to use the engine image next time I update unreal versions, I will be forced to maintain dockerfiles and build process for these images, or an engine fork. Adding these tasks to my ever-increasing workload would squash any hope of timely upgrading engine versions and force my team (and downstream) to use ancient technology.

Stripping the help text would be significantly faster and easier, but that's why I'm opening an issue instead of a pull request.

I know it's a lot to ask, but it would mean a lot if support is continued for the engine image going forward.

If nothing else, I hope this issue challenges the assumption many have that installed and source builds have feature parity (apart from the ability to modify the engine source code).

In my case, the performance cost of an engine image doesn't matter. I don't need to build this image every day, nor am I short of data storage. Most use cases don't require an engine image because functional differences are rare and performance is often a more motivating constraint. Further, the developers who use this project the most frequently switch engine versions or rebuild these containers.

[philip-klein]

After spending $10,000 in labor, we now have a workaround. I'd prefer we don't have to spend that every time we upgrade versions of unreal.

For curious minds, we ended up rolling a custom dockerfile by using the -layout option and modifying the results from there. Endless adjustments and fixes combined with lack of domain expertise resulted in a highly frustrating and lengthy process.

[slonopotamus]

I hope those $10,000 were spent on making a PR that restores engine image in ue4-docker?

After some discussion, we agreed that motivation of "nobody uses engine image" for #273 was wrong and there are cases when it is used. And the current understanding is that it would be good to restore it. However, I want to point out that engine image was broken for UE5 before removal.

[philip-klein]

Sadly no. We have a custom automation program we're adding to unreal so packaging a plugin can be done with multiple build targets at once. This custom automation tool is being injected into the source code of the build step instead of what we had previously: use the engine image and lay this extra code on top after the image was built.

If anything, I think we should be pushing for changes to unreal source for this.

[philip-klein]

If someone comes up with a way to write custom automation programs for unreal using only an installed build, I'd love to hear about it.

[philip-klein]

We didn't fix the engine image for UE5. We fixed our need for it.

[philip-klein]

I might spend some time fixing the engine target for UE5 in my free time. I think it would be nice to contribute back to the community.

[slonopotamus]

Completely untested and lacking UE4 support revival of ue4-engine image: #332

[adamrehn]

I'm a bit late to the party on this one, and ultimately I agree that reintroducing support for source builds is helpful if we have users who cannot use Installed Builds for one reason or another, but hopefully I can add a couple things of value here.

Building individual plugins with Installed Builds

Packaging individual plugins using AutomationTool's BuildPlugin script should function correctly in Installed Builds, and this has been supported for quite a while. The documentation on the Unreal Containers community hub that references doing so (which was written back in February 2020) was based on tests performed using Installed Builds.

As a sanity check, I've just tested packaging a simple plugin using the official Linux and Windows distributions of Unreal Engine 5.3.2, and packaging was successful under both platforms. Here's the command I used under Linux (under Windows, replace RunUAT.sh with RunUAT.bat, and Linux with Win64):

$ ./Engine/Build/BatchFiles/RunUAT.sh \
    BuildPlugin \
    -HostPlatforms=Linux \
    -TargetPlatforms=Linux \
    -Plugin="/home/adam/Documents/Unreal Projects/PluginTest/Plugins/TestPlugin/TestPlugin.uplugin" \
    -Package="/home/adam/Documents/Unreal Projects/PluginTest/dist"

@philip-klein if you're able to provide a minimal reproducible example of a plugin that fails to package with BuildPlugin under Installed Builds of the Unreal Engine, then I would be happy to pass that along to my contacts at Epic so they can investigate whatever underlying bug is causing the failures you've observed.

Using custom automation projects with Installed Builds

AutomationTool supports loading custom automation projects from a variety of locations, and although the Create an Automation Project page from the Unreal Engine documentation suggests that a source build is required for using custom automation projects, this is only true for automation projects that are located within the source tree of the Unreal Engine itself. Loading custom automation projects from arbitrary filesystem directories is actually supported in both source builds and Installed Builds of the Unreal Engine, and I've personally used custom automation projects with Installed Builds of Unreal Engine 4.27 and newer.

The easiest way to load a custom automation project with an Installed Build is by using AutomationTool's -ScriptDir flag. This flag allows you to specify an additional directory to be searched for *.Automation.csproj files, which will allow you to then use any automation scripts provided by that automation project.

Here's an example that I've just tested with the official Linux and Windows distributions of Unreal Engine 5.3.2:

• Create a directory to hold your custom automation project.

• Create an *.Automation.csproj file. The easiest way to create a project file that will work from outside of the Unreal Engine source directory is to take an existing project file from AutomationTool itself (e.g. AutomationScripts.Automation.csproj) and then replace any relative references to the engine's source tree with the placeholder $(EngineDir). You'll also need to give the project a unique assembly name, otherwise AutomationTool will treat it as a duplicate and ignore it.

  Here's my example project file, which I named MyProject.Automation.csproj:

<Project Sdk="Microsoft.NET.Sdk">
  <Import Project="$(EngineDir)\Source\Programs\Shared\UnrealEngine.csproj.props" />
  
  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
    <Configuration Condition=" '$(Configuration)' == '' ">Development</Configuration>
    <OutputType>Library</OutputType>
    <AppendTargetFrameworkToOutputPath>false</AppendTargetFrameworkToOutputPath>
    <GenerateAssemblyInfo>false</GenerateAssemblyInfo>
    <GenerateTargetFrameworkAttribute>false</GenerateTargetFrameworkAttribute>
    <Configurations>Debug;Release;Development</Configurations>
    <RootNamespace>AutomationTool</RootNamespace>
    <AssemblyName>MyProject.Automation</AssemblyName>
    <WarningsNotAsErrors>612,618</WarningsNotAsErrors>
    <OutputPath>.\Binaries\</OutputPath>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
    <DebugType>pdbonly</DebugType>
  </PropertyGroup>
  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Development|AnyCPU' ">
    <DefineConstants>$(DefineConstants);TRACE</DefineConstants>
    <Optimize>true</Optimize>
  </PropertyGroup>
  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
    <DefineConstants>$(DefineConstants);TRACE</DefineConstants>
    <Optimize>true</Optimize>
  </PropertyGroup>
  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
    <DefineConstants>$(DefineConstants);DEBUG;TRACE</DefineConstants>
    <DebugSymbols>true</DebugSymbols>
    <DebugType>full</DebugType>
  </PropertyGroup>

  <ItemGroup>
    <ProjectReference Include="$(EngineDir)\Source\Programs\Shared\EpicGames.Core\EpicGames.Core.csproj" PrivateAssets="All"><Private>false</Private></ProjectReference>
    <ProjectReference Include="$(EngineDir)\Source\Programs\AutomationTool\AutomationUtils\AutomationUtils.Automation.csproj" PrivateAssets="All"><Private>false</Private></ProjectReference>
    <ProjectReference Include="$(EngineDir)\Source\Programs\AutomationTool\Localization\Localization.Automation.csproj" PrivateAssets="All"><Private>false</Private></ProjectReference>
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.CSharp" Version="4.7.0" />
    <PackageReference Include="System.Data.DataSetExtensions" Version="4.5.0" />
    <PackageReference Include="System.Net.Http" Version="4.3.4" />
  </ItemGroup>
</Project>

• Create a C# source file for your custom automation script, and place it in the same directory as the project file.

  Here's my example script file, which I named MyScript.Automation.cs:

using AutomationTool;
using Microsoft.Extensions.Logging;

namespace AutomationTool
{
	[Help("Custom script.")]
	class MyScript : BuildCommand
	{
		public override void ExecuteBuild()
		{
			Logger.LogInformation("RUNNING MY CUSTOM SCRIPT!");
		}
	}
}

• Once the project file and script file are in place, invoke RunUAT.sh (RunUAT.bat under Windows) and specify both the -compile flag and the -ScriptDir flag. The -compile flag will ensure the assembly for the custom automation project is built, and is only technically needed when first running the script or when making changes to the source code, but it's a no-op if the assembly is already up-to-date, so it's safe to include it in every invocation.

  Here's the command I used to run my example under Linux, and the accompanying output:

$ ./Engine/Build/BatchFiles/RunUAT.sh \
    -compile \
    -ScriptDir="/home/adam/Desktop/MyAutomationProject" \
    MyScript

Running AutomationTool...

Fixing inconsistent case in filenames.
Setting up Mono
Setting up bundled DotNet SDK
Start UAT Interactively: dotnet AutomationTool.dll -compile -ScriptDir=/home/adam/Desktop/MyAutomationProject MyScript
/home/adam/Desktop/Linux_Unreal_Engine_5.3.2/Engine/Build/BatchFiles/Linux/../../../Binaries/ThirdParty/DotNet/6.0.302/linux/dotnet
Starting AutomationTool...
Parsing command line: -compile -ScriptDir=/home/adam/Desktop/MyAutomationProject MyScript
Initializing script modules...
ForceCompile not supported if Unreal.IsEngineInstalled() == true
Total script module initialization time: 0.10 s.
Executing commands...
RUNNING MY CUSTOM SCRIPT!
BUILD SUCCESSFUL
AutomationTool executed for 0h 0m 0s
AutomationTool exiting with ExitCode=0 (Success)

It's worth noting that older versions of the Unreal Engine would automatically append /Build to any path specified via the -ScriptDir flag, so it was necessary to nest the project and script files in a subdirectory named Build and then pass the parent directory to AutomationTool. However, this no longer appears to be the case in Unreal Engine 5.3, and my example worked as expected when using a flat directory structure for the source files.