XC_Engine [20] - XC_Core [7b] - XC_IpDrv

[SC]-[WARTZ_{HoF}]

I started doing some small reverse engineering on how map switch works, as well as how subsystems interacted with it and without realizing, stumbled upon something I wanted to fix years ago: traces going through movers on dedicated servers.
Continued to that, I kept making experiments with the GameEngine and got to achieve a few interesting things, things you'd see in a game patch now deployed into an extension.

***DOWNLOADS***

=========================================================

XC_Engine_20.7z
Server oriented update, deprecates AntiMsgHack
DOWNLOAD LINK

XC_Engine_19.7z
Requires XC_Core version 7
Function replacer, and path builder.
DOWNLOAD LINK

XC_Engine_18.7z
Requires XC_Core version 6.
Better memory management on servers.
Massive Linux fixes.
DOWNLOAD LINK

=========================================================
XC_Core - Base extension for native UT99 addons by Higor

Releases
7b: GIT UT99.org mirror

XC_Core_7.7z
Linux: only use with XC_Engine v19 or above
XC_Server commandlet with timer fix.
DOWNLOAD LINK

Code: Select all

Version 7b

=============
Installation:
=============
XC_Core.u
XC_Core.int
XC_Core.dll (win32)
XC_Core.so (linux)
LZMA.dll (win32)
LZMA.so (linux)
>>> ~UnrealTournament\System\


=================================
Setting up LZMA channel upload:
(optional, unredirected servers)
=================================
~UnrealTournament.ini
>>>
[IpDrv.TcpNetDriver] or [XC_IpDrv.XC_TcpNetDriver]
AllowDownloads=True
...
DownloadManagers=IpDrv.HTTPDownload
DownloadManagers=XC_Core.XC_ChannelDownload
DownloadManagers=Engine.ChannelDownload

Then keep the .LZMA (or .UZ) files on the same directory as the uncompressed versions.


=============================
LZMA Compression commandlets:
=============================

You can LZMA compress using a XC_Core commandlet:
UCC LZMACompress ..\Maps\CTF-Coret.unr

You can LZMA decompress using 7zip, WinRar or:
UCC LZMADecompress ..\Maps\CTF-Coret.unr.lzma

Both commandlets support wildcards.


===================
Additional natives:
===================
See XC_CoreStatics class (Object subclass).


===================
UBinary serializer:
===================
Now merged into XC_Core, proves minimal binary file handling to UnrealScript.
Check classes BinarySerializer (Object) and BinaryTester (Actor) for usage guidelines.

NOTES:
For security measures file writer doesn't allow creating files outside of the game directory.


======
Notes:
======
Having XC_Core files installed allows you to use LZMA channel download from servers that have this feature.
This package is required to run XC_Engine and XC_IpDrv.


========================
c++ headers and linking:
========================
This package contains headers that allow the user to utilize XC_Core features in own native packages.
Just add ..\XC_Core\Inc to include settings and link to XC_Core.lib (or XC_Core.so in Linux)

Don't forget to define this macro somewhere in your code (or preprocessor):
#define XC_CORE_API DLL_IMPORT

When reporting crashes, please attach the critical error logs if possible and tell what package/class is causing it.
=======================
XC_Engine - XC_GameEngine extension for UT99 by Higor.


===========
Setting up:
===========
** REQUIRES AT LEAST XC_CORE VERSION 7
Place XC_Engine binaries in your ~UT/System/ directory.
The new GameEngine we want to load has to be specified in UnrealTournament.ini (or it's server equivalent) as follows.

Code: Select all

[Engine.Engine]
;GameEngine=Engine.GameEngine
GameEngine=XC_Engine.XC_GameEngine
;NetworkDevice=IpDrv.TcpNetDriver
NetworkDevice=XC_IpDrv.XC_TcpNetDriver

Be adviced, when editing ServerPackages and ServerActors in a XC_Engine server, find the [XC_Engine.XC_GameEngine] entry!!!
Either remove it (and apply on GameEngine), or apply the changes on said (XC_GameEngine) entry instead.

Safe to use in v436-v451, and on ACE servers since most hacks are reverted during online sessions.
Just avoid AnthChecker servers until they have whitelisted this binary.


=================
Global features:
=================
- (Server, Standalone) Runtime UnrealScript/Native function replacer plus existing premade replacements (bugfixes, optimizations).
- (Conditional) Big collection of new native functions to use, check the UnrealScript source for documentation.
- (Linux) Server/Client communication no longer borks strings with non-standard characters.
- (All) Version 451 GET and SET command functionality + crashfix.
- (Server) Fake player and malformed string crash exploits fix, players cannot bypass password protection, massive log spam prevention.
- (Server) New Relevancy loop code that allows better/extended visibility conditions on actor replication, also fixes a few flaws on net priority.
- (Editor) New navigation network builder.
- (Client) Renderer-independant framerate limiter, combined with a fixed launcher can solve speed/timing issues: type "FPS [limit]".

- (Conditional) XC_Core natives have their numbered opcodes enabled for use without package dependancy.
- (Dedicated Server) Moving Brush Tracker in Dedicated servers, specific maps can be ignored.
- (Server) Ability to send 'undownloadable' maps.
- (All) Makes several properties from native only classes visible to UnrealScript, player commands and edit windows (win32). Below table for more info.
- (All) Timing fix for computers that change CPU frequencies (DOESN'T ADDRESS AMD TIMERS)
- (Server) TravelManager subsystem providing extended inventory handling for Coop games (don't lose items on reconnect).
- (Client) Automatic cache conversion to desired directories.
- (All) Collision Grid replacing the old hash, per-actor custom primitives support.
- (All) Cleaner map switch by nulling out transient actor references to the main level.
- (Server) Worked around RELIABLE_BUFFER crash on Windows servers.
- (Server, Win32) LZMA autocompressor can be run on a separate thread as soon as a map is loaded.
- (All) Lower memory usage and faster map cleanup on long games.
- (Server, experimental) Sliding player bug workaround.

Note:
(Conditional) Means
- (All) in Linux builds
- (Server, Standalone player) In Win32 builds, auto-disables itself when joining a server.

=================
XC_IpDrv
Enhanced Net Driver and file downloaders.
=================
Net Driver:
- ICMP unreachable exploit patched.
- Connection limit, kills dataless connections.
- (Experimental) Can connect to redirects via proxy.

HTTP LZMA file downloader.
- (Experimental) Can connect to redirects via proxy.


================
Extra commands.
Check other documentation files for more commands.
================
- EditObject Name=Objectname Skip=skipcount
Client, Win32 only.
Brings up a property editor dialog of an object with a specified name.
Skip= is optional and can be used to bring up a newer objects with said name.

Example: "EditObject Name=MyLevel Skip=1" Brings up play level's XLevel properties.
Example: "EditObject Name=MyLevel" Brings up Entry level's XLevel properties.

- FPS fpslimit
Sets internal framerate limiter between 4 and 200, lower than 4 means disable.

- DumpObject Name=Objectname
Dumps object in question's memory block into a file (with the object's name), only dumps the first object with matching name.
If the object is a UFunction, then it will also save a file name FUNCTIONDATA.bin with the script code (serialized TArray<BYTE>).

- LogFields Name=classname
Logs all of the UnrealScript visible properties of the specified class, with property flags, offset, size and array count.
Boolean properties have their bitmask info logged instead of array size.

- LogClassSizes Outer=packagename(optional)
Prints in log a huge list of classes and their size in memory.
If the Outer=packagename parameter isn't used (or fails), it will print all classes's sizes.

- ToggleTimingFix - TimingFix
Toggles the timing fix on/off.
Timing fix is enabled by default and is saved in [XC_Engine.XC_GameEngine] config entry.

- ToggleDebugLogs - DebugLogs
Toggles additional logging, for developers.
Disabled by default, saved in [XC_Engine.XC_GameEngine] config entry.

- ToggleRelevancy - ToggleRelevant
Requires bUseLevelHook.
Toggles XC_Level relevancy loop on net servers, see "Relevancy loop.txt" for details.

- AdminLoginHook actor_name
For code usage, lets an actor register itself as the AdminLoginHook actor (overrides previous one).

- OctreeDebug Name=actor_name
Prints FCollisionCacus information for an actor (if contained in the Octree system).

- TimeFactor
Displays the Time Manager's current time scaler, if active.
Values other than 1 (or approximate) indicate that XC_Engine is the one responsible
for keeping your game running at normal speed.


===================
Exposed properties:
===================
Additional properties are now visible on certain native classes and their subclasses, these increase the potential functionality of servers and clients running mods coded to access them via GetPropertyText() or GET commands.
See "Relevancy loop.txt" for extra properties in Actor.
= CLASS -> CPP_PropertyName -> UScript_PropertyName (type) (flags)

- GameEngine -> GLevel -> Level (Level) (const, editconst)
- GameEngine -> GEntry -> Entry (Level) (const, editconst)
- DemoRecDriver -> DemoFileName -> DemoFileName (string) (const, editconst)
- LevelBase -> NetDriver -> NetDriver (obj NetDriver) (const, editconst)
- LevelBase -> DemoRecDriver -> DemoRecDriver (obj NetDriver) (const, editconst)
- LevelBase -> Engine -> Engine (obj Engine) (const, editconst)
- LevelBase -> URL.Protocol -> URL_Protocol (string) (const, editconst)
- LevelBase -> URL.Host -> URL_Host (string) (const, editconst)
- LevelBase -> URL.Port -> URL_Port (int) (const, editconst)
- LevelBase -> URL.Map -> URL_Map (string) (const, editconst)
- LevelBase -> URL.Op -> URL_Options (array<string>) (const, editconst)
- LevelBase -> URL.Portal -> URL_Portal (string) (const, editconst)
- LevelBase -> Actors.Num() -> ActorListSize (int) (const, editconst)
- Level -> iFirstDynamicActor -> iFirstDynamicActor (int) (const, editconst)
- Level -> iFirstNetRelevantActor -> iFirstNetRelevantActor (int) (const, editconst)
- NetDriver -> ClientConnections -> ClientConnections (array<obj NetConnection>) (const, editconst)
- NetDriver -> ServerConnection -> ServerConnection (obj NetConnection) (const, editconst)

====================================
Functions patched/hooked in runtime:
====================================
See XC_Engine_Actor and XC_Engine_UT99_Actor for a full list of script patches.

Additionally this hooks still remains forced by internal code:
UWindowList.Sort -> Super fast, doesn't crash practice session when map count exceeds ~4000

=============================
Own XC_GameEngine properties:
=============================
- XC_Version (int) (const, editconst)
Stored in defaults, accesible via "GET INI:ENGINE.ENGINE.GAMEENGINE XC_VERSION"
Also accesible via "XC_Engine" console command.

- bHackingTracker (bool) (const, editconst)
Indicates that XC_GameEngine is handling the moving brush tracker.
Only true in Dedicated servers where the loaded map has movers.

- bDisableTimingFix (bool) (config) (default=false)
Disables timing fix.

- bDisableBrushTracker (bool) (config) (default=false)
Disables XC_Engine's addition of Brush Tracker.

- bSortMaplistByFolder (bool) (config) (default=false)
Sort the map cache alphabetically by folder.

- bSortMaplistGlobal (bool) (config) (default=false)
Sort the entire map cache alphabetically. This overrides the previous setting.

- bAutoTravelManager (bool) (config) (default=true)
TravelManager rebuilds the travel lists every two seconds.

- bCacheConvertAtJoin (bool) (config) (default=false)
Copy and rename cache files from current server into it's corresponding system paths.

- bFasterUpload (bool) (config) (default=true)
Forces a temporary NetSpeed of 1000001 on incoming clients for non-redirected downloads, reverts when finished.
The initial NetSpeed value is stored in ConfiguredNetSpeed during file download.

- bCollisionHashHook (bool) (config) (default=true)
Initializes levels using the FCollisionCacus octree system instead of the FCollisionHash.

- bEnableDebugLogs (bool) (config) (default=false)
Displays additional logs aimed at developers.

- NoBrushTrackerFix (array<string>) (config)
List of maps we want to exclude from using the brush tracker fixer (only works in dedicated server mode), ex (~UT.ini):
NoBrushTrackerFix=CTF-Niven
NoBrushTrackerFix=etc...

- bUseLevelHook (bool) (config) (default=true)
Hooks the level object after map load, allows it to execute XC_Engine code.

- bForceLevelHook (bool) (config) (default=false)
Forces level hook if net client.

- bUseNewRelevancy (bool) (config) (default=true)
Requires bUseLevelHook=True. See: - ToggleRelevancy - ToggleRelevant commands

- AdminLoginHook (actor) (const, editconst)
Actor handling AdminLogin requests, already passed the checks needed to function.

- bAutoCompressLZMA (bool) (config) (default=false)
Spawns a thread to LZMA compress every package loaded by the server.
To be used in conjunction with XC_Core's XC_ChannelDownload.

- bScriptDebug (bool) (config) (default=false)
Displays script callstack on crashlogs, useful for developers and server admins under attack.


=================
Credits:
=================
I would like to thank my fellow betatesters
- Chamberly
- ~V~
- Nelsona
- SC]-[LONG_{HoF} aka "SC]-[WARTZ_{HoF}"
- $carface (and the legions of Siege apes)
- AnthRAX
- SicilianKill

And all of Cham's development server visitors for the help in bugfixing this.
And to the website owners where I downloaded very educational sample codes of Unreal Engine 2 and 3 (lol!)

***ALTERNATE DOWNLOAD LINKS***

XC_Engine_20.7z

Server oriented update, deprecates AntiMsgHack

(429.51 KiB) Downloaded 746 times

XC_Engine_19.7z

Requires XC_Core version 7
Function replacer, and path builder.

(426.59 KiB) Downloaded 709 times

XC_Engine_18.7z

Requires XC_Core version 6.
Better memory management on servers.
Massive Linux fixes.

(371.83 KiB) Downloaded 668 times

XC_Core_7.7z

Linux: only use with XC_Engine v19 or above
XC_Server commandlet with timer fix.

(184.71 KiB) Downloaded 693 times

XC_Core_7b.7z

LZMA compression/decompression now dynamically loaded from LZMA library.
Changed LZMA dictionary size from 16mb to 4mb, memory usage for servers should be far lower during compression.
LZMACompress commandlet works in Linux.
*LZMA compression uses 1 core in Servers.
*LZMA compression uses 2 cores in win32 lzmacompress and 1 core in linux lzmacompress

(220.87 KiB) Downloaded 703 times

[Nelsona]

Current version of this extension is marked 21 according to logs - it do includes download quantity shown On Screen if player does use it.

[SC]-[WARTZ_{HoF}]

I went to the head of Higor's thread. He apparently hasn't included this v21 on first page yet.

[Nelsona]

Bump here. I must be agree here with you, Sir. I was updating today the server with MH-Detour map and required packages. Even if All did work in high speed, something was strange with this v21 so I went to check logs from redirect and server. Simply client went to duplicate file download. Client has downloaded from redirect then from server lzma files, and then again uncompressed files. By restricting downloads from server and leaving only redirect, client has downloaded file from redirect and then being rejected for not having file, LOL...
This package count + the redirect deal seems borked since the new "Redirection" deal. Actually client is even ignoring UseCompression directives and is simply downloading 3 files: 2 compressed and one directly sent uncompressed. Probably the best XC Server deal is NO Redirect and NO lzma for good. At this moment lzma content it's only a waste of HDD space, randomly player doesn't seems to use it - I'm speaking about last v21 if I'm not mistaking. V20 for me was way better at this point so I think I'll go back to v20, of course I have to figure everything what is happening and when is happening.
Edit:
These were happening if player has some XC files in system and IpDrv driver as default vanilla player. If these are not active they should not mess the downloads. If player is messing the config in some servers might be served with a total rejection even if previously has already downloaded some files from redirect. Returning to server for a Duplicate download doesn't make any sense.

[SC]-[WARTZ_{HoF}]

Got a pm from chamberly. These redirect issues you are talking about with v21 look like these that she sent me?

She Said:

Got a redirect/download issue, got crash from both XC and regular IPDrivers.

[Nelsona]

Some note around.
You have a default UT install but you have plans for usage of XCGE. By default you have to create a folder where you will have all "XC_..." related files. If you have modified INI for XCGE drop XC files in UT. If you have a problem and you want to disable XCGE here is a Warning, after editing INI back to normal, DO NOT LEAVE XC files there or funky stuff will occur - move them back into warehouse outta UT in mentioned folder. If you have XC files but not declared in INI, they can be triggered by a potential XC server and you will have multiple downloads. If by chance admin has disabled direct downloads (because has redirect) you might be rejected permanently. Actually XC_IpDrv doesn't notice engine about file which was already downloaded and player thinks that has no files, this deal if is not solved I recommend when disabling XCGE to remove XC files as well, else is a waste of time to have multiple downloads of the same file.
For me age of UZ is ended.
Vanilla player has Normal new files from Engine.ChannelDownload VIA XCGE. XC player has lzma from redirect with bandwidth restricted at 300 kb/s - suffice for shorter lzma files.
Moron player (me at random) disables XC_Ipdrv but keeps XC_Engine loaded and XC files = Triple Download: 1 lzma from redirect, 2nd lzma from server, 3rd uncompressed from server - this is a BAD move, XC_IpDrv after downloading files simply doesn't notice Engine about an already executed download, and it will download that file again.

Note 2:
And now I think this download task - UZ on purpose - has a buffer which has some limitation, not sure if has something to do with that "cache" setup but it looks like exist some variable called
"OverrideBufferAllocation" which I'm setting up to True because after my logic if buffer is too small it will need to be increased for big-ass files. I'm not sure if this is available in all versions but I'll try to figure for 440, 436 if IpDrv has such thing (and XC_IpDrv too).

I'll be back if I have data. Higor should know these better but if he has decided to not chat that much, then I will need a bit of time for figuring these. It might be needed to make a config reporter actor for logging all things... l'll look around.

Maybe Feralilaref or nogardidragon knows these somehow...

Edit:
I triggered some console Commands. It looks like buffer doesn't exist in XC, that's IpDrv deal. Else XC_IpDrv is like None if a network game is not running. It responds in console only in case of a connection ignited to a server. So probably that error is from programming if setup is OK.
Anyway I did not see it, I'm not using UZ so I cannot figure any issues at this point in my test server.

Oops me: Right now I see "_XC" suffix - this is Higor's fault...

[SC]-[WARTZ_{HoF}]

Well hof servers use default IpDrv plus xcgev20 because we have UZ redirect. I didn't see a reason to use lzma since bandwidth is capped at 6mb at my box. even if lzma is better for compression having 20 clients downloading those files at same time from my box is not good for other player pings already ingame.

[Nelsona]

Yes, direct download is not a bless for 6 mb... Need bandwidth... else redirect should have 2 compressions = more space wasted...

[SC]-[WARTZ_{HoF}]

If lzma compression could be pulled from the fast redirect rather than the server box locally then that would be great. But you still also have the issue that not all clients know or are willing to use xcge client side.

[Nelsona]

There is an alternate option - the evil one - if compressor has flaws redirect can be done without compression, downloading identical files with server but no uz lzma used. If Epic did that bool as option for disabling compression on demand perhaps they had some reason (known somehow...) for it.