虚幻热重载代码

热重载代码探究

Editor打开同时编译C++代码,UE可以支持代码模块热重载, 这个特性挺神奇, 今天来探究下 .

基本推测

• Editor 监听事件(例如socket,或者modules之类的表示模块版本的文件), 收到事件时, 主动卸载旧模块, 加载新模块 .

• 新模块加载后, UE的对象重新注册(地址)和初始化.

• 新模块加载后, 原来过程中的对象地址可能发生变化, 如果故意不符合UE模块规范和内存规范暴漏的地址, 在其他模块中访问可能引起崩溃。

求证

实际实验来观察和推测的不一致

Editor模式编译工程C++

可以看到日志有出现: Starting Hot-Reload from IDE

分析代码

根据字符串提示, 找到代码中的相关模块和函数

void FHotReloadModule::DoHotReloadFromIDE(...)
{
    ...
    TArray<FString> GameModuleNames = UEHotReload_Private::GetGameModuleNames(ModuleManager);
    if (GameModuleNames.Num() > 0)
    {
        FScopedDurationTimer Timer(Duration);

        if (NewModules.Num() == 0)
        {
            return;
        }

        UE_LOG(LogHotReload, Log, TEXT("Starting Hot-Reload from IDE"));

        HotReloadStartTime = FPlatformTime::Seconds();

        FScopedSlowTask SlowTask(100.f, LOCTEXT("CompilingGameCode", "Compiling Game Code"));
        SlowTask.MakeDialog();
        ...
    }
    ...
}
...
ECompilationResult::Type FHotReloadModule::DoHotReloadInternal(...){
    ...
        CollectGarbage(GARBAGE_COLLECTION_KEEPFLAGS); // we create a new CDO in the transient package...this needs to go away before we try again.

// Abandon the old module.  We can't unload it because various data structures may be living
        // that have vtables pointing to code that would become invalidated.
        ModuleManager.AbandonModuleWithCallback(ShortPackageFName);

        // Load the newly-recompiled module up (it will actually have a different DLL file name at this point.)
        bReloadSucceeded = ModuleManager.LoadModule(ShortPackageFName) != nullptr;
        if (!bReloadSucceeded)
        {
            HotReloadAr.Logf(ELogVerbosity::Warning, TEXT("HotReload failed, reload failed %s."), *PackageName);
            Result = ECompilationResult::OtherCompilationError;
            break;
        }
    ...

}

从代码逻辑可以看到,在监测到 模块变化时, 清理模块路径缓存相关,执行GC, 然后对于变更的模块,执行AbandonModuleWithCallback, 然后重新加载模块, 具体加载流程是：
找到DLL并找到DLL中的InitializeModule函数(符号), 调用之, 在这个函数中会注册自身的UObject信息(UPackage), 返回了模块的IModuleInterface 接口对象, 然后再执行到 StartupModule 启动模块。加载完成之后对于依赖的模块也要执行卸载和加载的操作, 最后再对所有的UObject中UFunction的新函数地址, UScriptStruct做一些修正设置,然后再次执行GC清理 。

关键代码:AbandonModuleWithCallback

Module->Module->PreUnloadCallback();

AbandonModule( InModuleName );

// Ensure module is unloaded
check(!IsModuleLoaded(InModuleName));

这段代码主要是调用了模块的PreUnloadCallback, 然后执行了AbandonModule

// Will offer use-before-ready protection at next reload
ModuleInfo.bIsReady = false;

// Allow the module to shut itself down
ModuleInfo.Module->ShutdownModule();

// Release reference to module interface.  This will actually destroy the module object.
// @todo UE4 DLL: Could be dangerous in some cases to reset the module interface while abandoning.  Currently not
// a problem because script modules don't implement any functionality here.  Possible, we should keep these references
// alive off to the side somewhere (intentionally leak)
ModuleInfo.Module.Reset();

// A module was successfully unloaded.  Fire callbacks.
ModulesChangedEvent.Broadcast( InModuleName, EModuleChangeReason::ModuleUnloaded );

可以看到主要是调用了模块的关闭 是 ShutdownModule接口 。

值得注意的是，实际上并没有将模块DLL卸载Unload, 前面的注释表明,
可能其他模块还有以前的地址引用, 直接卸载DLL可能导致非法访问 。
所以应该Editor 每次编译后进程的内存都会增长一些, 但不会太多, 因为一般来说修改的模块并不会太大.

关键代码:ModuleManager.LoadModule

// Make sure that any UObjects that need to be registered were already processed before we go and
    // load another module.  We just do this so that we can easily tell whether UObjects are present
    // in the module being loaded.
    if (bCanProcessNewlyLoadedObjects)
    {
            ProcessLoadedObjectsCallback.Broadcast(NAME_None, bCanProcessNewlyLoadedObjects);
    }

    // Try to dynamically load the DLL

    UE_LOG(LogModuleManager, Verbose, TEXT("ModuleManager: Load Module '%s' DLL '%s'"), *InModuleName.ToString(), *ModuleInfo->Filename);

    if (ModuleInfo->Filename.IsEmpty() || !FPaths::FileExists(ModuleInfo->Filename))
    {
        TMap<FName, FString> ModulePathMap;
        FindModulePaths(*InModuleName.ToString(), ModulePathMap);

        if (ModulePathMap.Num() != 1)
        {
            UE_LOG(LogModuleManager, Warning, TEXT("ModuleManager: Unable to load module '%s'  - %d instances of that module name found."), *InModuleName.ToString(), ModulePathMap.Num());
            OutFailureReason = EModuleLoadResult::FileNotFound;
            return nullptr;
        }

        ModuleInfo->Filename = MoveTemp(TMap<FName, FString>::TIterator(ModulePathMap).Value());
    }

    // Determine which file to load for this module.
    const FString ModuleFileToLoad = FPaths::ConvertRelativePathToFull(ModuleInfo->Filename);

    // Clear the handle and set it again below if the module is successfully loaded
    ModuleInfo->Handle = nullptr;

    // Skip this check if file manager has not yet been initialized
    if (FPaths::FileExists(ModuleFileToLoad))
    {
        ModuleInfo->Handle = FPlatformProcess::GetDllHandle(*ModuleFileToLoad);
        if (ModuleInfo->Handle != nullptr)
        {
            // First things first.  If the loaded DLL has UObjects in it, then their generated code's
            // static initialization will have run during the DLL loading phase, and we'll need to
            // go in and make sure those new UObject classes are properly registered.
                    // Sometimes modules are loaded before even the UObject systems are ready.  We need to assume
                    // these modules aren't using UObjects.
                        // OK, we've verified that loading the module caused new UObject classes to be
                        // registered, so we'll treat this module as a module with UObjects in it.
                ProcessLoadedObjectsCallback.Broadcast(InModuleName, bCanProcessNewlyLoadedObjects);

            // Find our "InitializeModule" global function, which must exist for all module DLLs
            FInitializeModuleFunctionPtr InitializeModuleFunctionPtr =
                (FInitializeModuleFunctionPtr)FPlatformProcess::GetDllExport(ModuleInfo->Handle, TEXT("InitializeModule"));
            if (InitializeModuleFunctionPtr != nullptr)
            {
                if ( ModuleInfo->Module.IsValid() )
                {
                    // Assign the already loaded module into the return value, otherwise the return value gives the impression the module failed load!
                    LoadedModule = ModuleInfo->Module.Get();
                }
                else
                {
                    // Initialize the module!
                    ModuleInfo->Module = TUniquePtr<IModuleInterface>(InitializeModuleFunctionPtr());

                    if ( ModuleInfo->Module.IsValid() )
                    {
                        TRACE_CPUPROFILER_EVENT_SCOPE_TEXT(*(InModuleName.ToString() + TEXT("::StartupModule")));

                        // Startup the module
                        ModuleInfo->Module->StartupModule();
                        // The module might try to load other dependent modules in StartupModule. In this case, we want those modules shut down AFTER this one because we may still depend on the module at shutdown.
                        ModuleInfo->LoadOrder = FModuleInfo::CurrentLoadOrder++;

                        // It's now ok for other threads to use the module.
                        ModuleInfo->bIsReady = true;

                        // Module was started successfully!  Fire callbacks.
                        ModulesChangedEvent.Broadcast(InModuleName, EModuleChangeReason::ModuleLoaded);

                        // Set the return parameter
                        LoadedModule = ModuleInfo->Module.Get();
                    }
                    else
                    {
                        UE_LOG(LogModuleManager, Warning, TEXT("ModuleManager: Unable to load module '%s' because InitializeModule function failed (returned nullptr.)"), *ModuleFileToLoad);

                        FPlatformProcess::FreeDllHandle(ModuleInfo->Handle);
                        ModuleInfo->Handle = nullptr;
                        OutFailureReason = EModuleLoadResult::FailedToInitialize;
                    }
                }
            }
            else
            {
                UE_LOG(LogModuleManager, Warning, TEXT("ModuleManager: Unable to load module '%s' because InitializeModule function was not found."), *ModuleFileToLoad);

                FPlatformProcess::FreeDllHandle(ModuleInfo->Handle);
                ModuleInfo->Handle = nullptr;
                OutFailureReason = EModuleLoadResult::FailedToInitialize;
            }
        }
        else
        {
            UE_LOG(LogModuleManager, Warning, TEXT("ModuleManager: Unable to load module '%s' because the file couldn't be loaded by the OS."), *ModuleFileToLoad);
            OutFailureReason = EModuleLoadResult::CouldNotBeLoadedByOS;
        }
    }
    else
    {
        UE_LOG(LogModuleManager, Warning, TEXT("ModuleManager: Unable to load module '%s' because the file '%s' was not found."), *InModuleName.ToString(), *ModuleFileToLoad);
        OutFailureReason = EModuleLoadResult::FileNotFound;
    }

这段代码实际上属于ModuleManager , 就是一个加载模块的流程 .
主要就是加载DLL (非Monolithic), 然后调用其中的注册接口(也就是UHT Generated 中注册的UObject反射信息), 获得IModuleInterface后调用StartModule 启动模块。

关键代码:UnloadOrAbandonModuleWithCallback

对于变更模块依赖的模块, 也要重新加载 .

    // Load dependent modules.
    for (FName ModuleName : InDependentModules)
    {
        if (!ChangedModules.Contains(ModuleName))
        {
            continue;
        }

        ModuleManager.UnloadOrAbandonModuleWithCallback(ModuleName, HotReloadAr);
        const bool bLoaded = ModuleManager.LoadModuleWithCallback(ModuleName, HotReloadAr);
        if (!bLoaded)
        {
            HotReloadAr.Logf(ELogVerbosity::Warning, TEXT("Unable to reload module %s"), *ModuleName.GetPlainNameString());
        }
    }

...


void FModuleManager::UnloadOrAbandonModuleWithCallback(const FName InModuleName, FOutputDevice &Ar)
{
    auto Module = FindModuleChecked(InModuleName);
    
    Module->Module->PreUnloadCallback();

    const bool bIsHotReloadable = DoesLoadedModuleHaveUObjects( InModuleName );
    if (bIsHotReloadable && Module->Module->SupportsDynamicReloading())
    {
        if( !UnloadModule( InModuleName ))
        {
            Ar.Logf(TEXT("Module couldn't be unloaded, and so can't be recompiled while the engine is running."));
        }
    }
    else
    {
        // Don't warn if abandoning was the intent here
        Ar.Logf(TEXT("Module being reloaded does not support dynamic unloading -- abandoning existing loaded module so that we can load the recompiled version!"));
        AbandonModule( InModuleName );
    }

    // Ensure module is unloaded
    check(!IsModuleLoaded(InModuleName));
}

这里做了检查, 如果模块已经加载(并有UObject)并且支持动态重载SupportsDynamicReloading 的情况下,会调用UnloadModule先卸载模块,否则同之前的变更模块一样只有 AbandonModule 。

其中Unload 也是ModuleManager实现的功能, 内部主要也调用了模块的Shutdown和 FreeDllHandle (当确认需要释放时,默认也不调用) 平台接口

关键流程:重载成功后处理

遍历系统所有UObject 针对:

• UFunction
• UScriptStruct

重新设置UFunction的Native地址
将新的UPackage中的ScriptStruct收集起来修改相关属性

监听模块变更事件

void FHotReloadModule::AddHotReloadDirectory(IDirectoryWatcher* DirectoryWatcher, const FString& BaseDir)
{
    FString BinariesPath = FPaths::ConvertRelativePathToFull(BaseDir / TEXT("Binaries") / FPlatformProcess::GetBinariesSubdirectory());
    if (FPaths::DirectoryExists(BinariesPath) && !BinariesFolderChangedDelegateHandles.Contains(BinariesPath))
    {
        IDirectoryWatcher::FDirectoryChanged BinariesFolderChangedDelegate = IDirectoryWatcher::FDirectoryChanged::CreateRaw(this, &FHotReloadModule::OnHotReloadBinariesChanged);

        FDelegateHandle Handle;
        if (DirectoryWatcher->RegisterDirectoryChangedCallback_Handle(BinariesPath, BinariesFolderChangedDelegate, Handle))
        {
            BinariesFolderChangedDelegateHandles.Add(BinariesPath, Handle);
        }
    }
}

这里就是FHotReload 模块使用DirectorWatcher监测目录的文件变更,然后判定变化触触发HotReload 操作的地方了。

结论

大体同推测一致, 主要是细节处理, 对于变更模块并不能卸载Unload, 只是调用Shutdown相关