The end of an era…

Next week is WWDC, the hyped and supposedly exciting event for Apple Developers! New versions of macOS, iOS, tvOS and watchOS. New API’s. New things to take advantage of. Except this year I can’t help but feel down…

Last year Apple made it clear that macOS Mojave would be the end of the line for 32-bit applications. This means that the next version of macOS will have stripped out all 32-bit functionality from the Kernel and any frameworks that still have it, and most likely Carbon will be finally put to rest.

From the point of view as a developer, this is great! Refinement, cleaning up and optimisation in the tools I use everyday is a great thing and makes my job easier. As someone who enjoys playing some older games, such as EV Nova, this is not so great.

The latest versions of the EV Nova binary itself only contains code for the i386 architecture (32-bit intel). Earlier version also contain 32-bit PowerPC, but thats even more obsolete! From now on it looks like if I ever want to play EV Nova I’ll be firing up a VM in order to do so.

When I first began investigating the ResourceFork format back in 2014, I was aware that Carbon was already old news and utterly deprecated. I had hoped to be able to complete the entire EV Nova clone project over the course of a few years. I had not really anticipated how much other events in my life would begin to consume my time, thus causing the project to take longer.

Now here we are, more or less the end of the road for the technologies of the old world, and I’ve not been able to get the game completed or really even started in any meaningful capacity. I just have a bunch of research and information on those old formats and technologies.

Forging a new path

Over the past week I have been thinking (arguably a dangerous thing) about the future direction of the project. The point of this whole thing, and this project is to allow EV Nova to survive. Immortalised as an Open Source project that can be updated when required.

This means I need to do it right. I need to make it future proof. I need to make it portable, and detached from core technologies of any given operating system. So I’m going back to ground zero. I’m going to rebuild ResourceKit and ClassicKit. I’m going to introduce another framework NovaKit, which will be responsible for handling EV Nova resources and the likes. Each of these will be built in Swift using the package manager allowing them to be easily ported to Linux or Windows.

They will then be used by the main engine which will also be Swift, using the package manager again to provide and construct the core functionality of the game, sans any kind of graphical output. This is to ensure it has no platform specific functionality.

Finally a “rendering layer” will provide specific functionality for producing the games output and input for a given platform. Different platforms would include different rendering layers, i.e. macOS might make use of SpriteKit or Metal, whilst Linux might make use of OpenGL, and DirectX for Windows.

Player Data – The NpïL resource

One of the more irritating resources to deal with is the player data resource, or the NpïL resource. Before we get into how this resource can be read, we need to first understand some of the complexities and issues with it.

For whatever reason AmbrosiaSW never migrated this particular file to flattened ndat files. This means that the actual NpïL resource is inside a true resource fork. This is illustrated by the fact that when we drop the file in to a hex editor, we see no content at all, despite Finder reporting an 87KB file size.

This is a problem as none of the Cocoa or POSIX API’s can open or read from the resource fork data. The only API’s that can are the Carbon API’s. Thankfully the macOS virtual file system provides us a “hack” to read the resource fork as a data fork (though who knows for how much longer). We’ll take a look at this later.
The second issue is that the resource data itself is encrypted. There is a document online covering the actual data layout of the pilot resources (I believe by guy). However I can not currently find it, so not able to link to it. I have attached two screenshots beneath of the code structures representing the two NpïL resources.
NpïL – 128 “Pilot Data” – structure

NpïL – 129 “Ship Name” – structure

Yep, both resources have a different format internally, which means they can’t be read with exactly the same process. This is rather annoying. On top of that, if we parse the resources using these structures the data is completely garbled and insane due to the encryption. We’ll get into the decryption in a bit.

The Contents of a Pilot File

Screenshot 2019-05-27 at 07.51.47.png — Good old ResEdit and Mac OS 9 to the rescue

Nothing on modern macOS will allow us to look at the contents of a resource fork easily. It just doesn’t exist. The API’s required were deprecated 10 years ago, and do not function correctly under 64-bit.

This means that liberal use of SheepShaver (a PPC emulator, for Classic Mac OS) is required. As you can see this is not easy to actually look at… even unencrypted binary data is not easy to look at, but this is especially not easy to look at. As far as I can tell there is zero documentation about this encryption online, or at least zero surviving documentation. This means that the only way to deal with it was by reverse engineering EV Nova itself.

Screenshot 2019-05-27 at 08.11.57.png — The routine inside EV Nova responsible for Loading Pilot Data

This is a Hopper Disassembler, and it is used for investigating the contents of application binaries. I’m using it to reverse engineer parts of EV Nova itself. In addition to finding the _LoadPilotData routine, I also ran it through Hopper’s decompiler. The result of which can be found here.

The actual encryption algorithm is listed below. Being the result of a disassembly and decompilation, it is annoyingly cryptic and difficult to follow.

Screenshot 2019-05-27 at 08.21.34.png — EV Nova’s pilot encryption algorithm

To be fair, I’ve written the code in much the same way many of the original variables are gone thanks to the original compiling of EV Nova. That’s fine, we don’t technically need them, they just make it easy to read.

The 3 arguments in the function are as follows:

arg0: The data to encrypt/decrypt
arg1: The size of the data to encrypt/decrypt
arg2: The encryption key (which is hex value 0xb36a210f)

So once you have all of this sorted out and resolved you are ready to read the contents of a Pilot File!

Not quite. Pilot resources aren’t in ndat files for some reason.

The ResourceFork under modern macOS

There is something fundamentally wrong with this, and it honestly surprised me. Why were pilot files not migrated to ndat files? My assumption is due to the existing pilot files out in the world and the desire to not render them dead and forgotten. That said I’m not sure that argument would have been valid as the same resource loading API’s are used for both ndat and pilot files alike.

As I mentioned earlier it is impossible to load a resource file using modern none deprecated API’s. Carbon provides the only way to do this, and it is 32-bit only. It will not work under 64-bit.

However the macOS virtual file system does provide a means of reading resource fork data into a separate data file, or via a pipe.

$ /path/to/resource-fork/..namedfork/rsrc

By appending the /..namedfork/rsrc to a file, you can read its ResourceFork.

Let’s return to Hex Fiend, and attempt to load the pilot file again, but using this command instead.

Screenshot 2019-05-27 at 08.36.33.png — At the bottom you can see the ship name!

It all worked successfully. Armed with all these tools, it is possible to fully read and handle pilot files from the original EV Nova engine.

An update on very slow progress

So it’s been quite a while since my last update on here. That said I’ve been posting a little more frequently on the /r/evnova subreddit. The last year or so has been hectic, leaving me with very little time to work on this project (moving, renovating, and my partner and I are expecting our first child very soon).

That said I do work on this whenever I can, or have the motivation to try and conquer some of the stubborn formats used. I’m now getting closer and closer to having everything parsed, and I think I have all of the documentation required to do so.

Since my last post I have managed to determine/decode the following formats (of which I’ll provide some documentation for soon.)

The character file encryption. The npïl resource was never heavily documentation to begin with, but information does exist in a vanishingly few places. However at some point an encryption was added to the npïl data making it impossible to easily edit as before. I’ll make a post about this soon.
The pattern resource, or ppat is used to provide the static on the status bar in the mini map. It’s rather annoying because it probably could have been provided as anything else and be easier to work with… well back then I suppose API’s existed to do just that!
The sound resource, or snd, is a cocktail of varying formats and the the extension ‘snd’ has been used for so many different file formats over the years that it is almost impossible to determine how to decode this thing. I’ve found old Apple Documentation about ‘Type 1’ and ‘Type 2’ sounds, but they are so limited on details. Hopefully it gives a starting point for writing a full parser.
The color icon, or cicn is another complex format that I just need to get done. I have documentation for it as with ppat and snd, but it’s high level details and doesn’t go into the exact structure exactly.
DLOG/DITL. These are used to define window layouts and Nova uses them to define the layout of windows such as Shipyard, Mission Computer, Landing Screen, etc… all of them in fact.

Well that’s all for now. Hopefully I’ll begin to have more things to show soon.

Starting on the Engine

This is by no means a full engine yet. It handles the opening sequence and gets the main menu to this state. Utterly useless to a player of the game. However this represents a major milestone.

For this screen to be rendered properly using nothing but the Nova data files, it needs to be able to decode rlëD and PICT resources, as well as understand bespoke Nova resources such as cölr and spïn. There are still a lot of other resource types that need parsing to make the engine complete. Not to mention replicating all the game logic!

I’m building the game in SpriteKit. This means that it should be possible to get it to run on macOS, iOS and tvOS all with the same basic codebase, although control systems are going to be more problematic. EV Nova’s built in system only really works with a Mac.

I think I’m probably going to make the UI (things like interaction, new pilot, settings, etc) be a little more friendly to touch environments on iOS, whilst leaving it as is on macOS.

Run Length Encoding – RLË Resources

EV Nova encodes all of its sprites in the RLË format. Specifically split between rlë8 and rlëD in the Nova engine. The 8 and D correspond to the colour/bit depth of the sprite images. For modern computers we’re only ever going to be concerned about rlëD.

RLË as it turns out is very similar, but not as complex, as the PICT format. It uses a series of opcodes to instruct the decoder on how to product an image. Luckily, it turns out I made a crude prototype of one quite a while ago. All I had to do was update the code and it worked.

The Starbridge. The icon of EV Nova, literally.

Well as it turns out, we’re now loading in all graphical assets except for the CICN and PPAT assets. Everything is just game data.

An update on the QuickDraw and Pictures

It turns out PICT is one of those formats that is just crap. It’s needlessly complex and for every way of encoding something, there is potentially 2 or 3 others ways of doing so as well. On top of that, some areas of the spec seem to be vague, repetitive or contradictory and just there to make life hell.

I’m not a fan of this format at all.

Why the rant?

So it turns out EV Nova’s data files use the PICT format for many things. UI graphics, targeting graphics, ship icons, landing screens, etc. Lots of them. However there currently appear to be 3 types of PICTs used. I’ve accounted for two of them, but the final one is infuriating, as finding documentation or reasons is impossible. I doubt this is the fault of the original designers. It was the software they used that did the encoding.

Single pixel width images are the problem. These are used by the middle graphics of buttons in the UI. These fail to parse properly.

On the bright side…

So as I’ve been searching for more information about this format, I have been able to find something interesting thankfully.

This is buried in the depths of the system, nested inside other frameworks. It’s the QuickDraw framework. In which all the functionality for QuickDraw and it’s format exists. This framework is deprecated and use of it is not really recommended these days. It will limit the app to 32-bit only and will not handle retina displays correctly.

But it does mean I have a reference point. How?

QuickDraw – _DrawPicture in Hopper Disassembler

Hopper Disassembler! A great tool for peeking inside and inspecting compiled binaries. That QuickDraw framework includes a binary that I am able to pull apart and study. Hopefully some of the secrets of the Picture resource will begin to reveal themselves.

The ResourceFork – A file of files.

The ResourceFork. A relic from a bygone age of the Mac. It was an elegant solution to the problem of software resources. All of the assets and resources for a file or program would be contained within the ResourceFork. To the user this would then appear as a single file on the file system.

However it had issues. If you tried to move a file that had a resource fork to a machine that used a different file system, or just plain didn’t understand them then you would run into trouble. The ResourceFork of the file would be truncated and you were left with an empty file. Oops. This was particularly common when sending files over the internet or transferring them on USB Thumb Drives to a Windows machine.

Apple has since supplanted the ResourceFork with the concept of the “Bundle”, a folder that can appear to the user as a single file. Mac OS X Applications are bundles, and contain a number of files inside related to the Application.

In fact, Apple supplanted the ResourceFork so long ago, that it has never been a first class citizen on Mac OS X. That’s nearly 17 years since it was effectively “deprecated”. Given that it was a predominantly a feature from the classic era of Mac OS, the API’s have lived on in the Carbon framework (which is on the way out currently). The actual functionality of the ResourceFork is now just a consequence of HFS+, which is itself being replaced by the newer file system, AFS.

Luckily EV Nova’s resource forks were flattened into data forks and live on as ndat files or rez files. However huge scores of plugins out there are probably still original ResourceForks.

But the ResourceFork situation is the case with everything in EV Nova. The ResourceFork’s house a number of formats which are all also deprecated and becoming relics of a bygone era. Functionality is fast disappearing. At some point of Apple will release the ultimate death blow to EV Nova, and old classic era software. They will remove Carbon and HIToolbox from the system. When that day arrives EV Nova will lose the ability to read anything from its data files.

That makes this project more than just a quest to recreate the engine for modern machines. It makes it an endeavour to preserve these old formats from being lost to the sands of time.

GitHub Project Page

Who doesn’t love some open source software? I’ve put up the project on GitHub so that all those interested in browsing the code, contributing or just keeping track of progress can see it all happening there.

http://github.com/tjhancocks/OpenNova

QuickDraw and the Picture Resource

Anyone that has worked with me will know that I love low level and/or old technology. So when you mix the two together, I am in my element! QuickDraw is an old, low level library for handling media. It first appeared on the earliest versions of the Macintosh System Software. It should have been right up my street.

Archaic Formats

It turns out old technology is not always documented well. In some cases it’s not really documented at all. Once upon time QuickDraw was very well documented. Well it was well documented from an API standpoint. The internals were… well there was some information about how it worked, but precious little. A document does still exist online for it. It’s in the archived section of Apple’s old websites. It’s basically a PDF of a scanned document. Not fantastic, but better than nothing.

QuickDraw it seems went through several iterations. Thankfully, by the time the Escape Velocity series came on to the scene. Version 2 of QuickDraw the default, and thus all picture resources appear to be encoded as such. This is both a blessing and curse.

Version 2 is the only one that I’ve had any luck in finding documents for. Even then there isn’t much. That document mentions a tiny amount about Version 1. The only thing I could learn was, that Version 1 was basically easier to parse. Damn. That means more work in trying to get the picture decoder up and running.

The Challenges

There are 2 major difficulties with the Picture Resources in QuickDraw. These are of course only the challenges that I’ve encountered. It turns out that EV Nova data files only seem to use a thin slice of the Picture Resource functionality.

Op-codes
Yes. The Picture Resource uses a series of op-codes to encode data. This is done primarily so that pictures can be encoded as a type of vector graphic. This is pretty cool considering when the format was designed. However there is only really one op-code that is needed for EV Nova. That is the Draw Direct Packed Bits operation. This opcode basically reads a massive chunk of data from the resource. This data is an encoded bitmap.
PackBits
This is an algorithm that is used to compress/pack bitmap information into a smaller space. The exact algorithm is notoriously difficult to find online and there appears to be multiple variants of the algorithm.

On top of those issues, the colour space for the original pictures was RGB 555, which is no longer a ready defined colour space on the Mac. So the bitmap information once unpacked needs to be translated into RGBA 8888.

When it goes wrong…

When decoding a QuickDraw picture goes wrong.

There is so much minutia in the format that it is extremely common to end up with results that appear something like you can see above, or worse. A lot of images will still fail to parse correctly due to some unexpected opcodes that I’ve yet to handle or decipher.

And this is just the picture resource. There is still the cicn, rlë, ppat, and other QuickDraw formats to handle.

OpenNova. Rebuilding a classic Mac OS game.

A few months ago I decided to undertake a slightly ludicrous side project. To re-implement a modern version of the classic Mac OS game, Escape Velocity Nova. The game is at this point essentially abandoned. Ambrosia Software is but a ghost of shell of their former selves and the game receives very little in the way of support. Furthermore it also relies on technologies from the Classic Macintosh days such as ResourceForks, QuickDraw and various others. Apple are advancing macOS each year and its becoming apparent that a lot of the old classic games are at the end of their lives.

EV Nova already requires an unofficial patch (or tweaking of the application binary) to run on systems later than macOS 10.11. Sooner or later, Apple is going to deliver the final nail to the coffin.

This is where my idea was spawned. I had once been a member of the community for EV Nova (particularly on ev-nova.net). I loved the game and would play it constantly. I was much younger at the time (true for everyone, unless you’re Benjamin Button) and had just started to really learn how to develop software. I decided to try my hand at creating a plugin development tool for EV Nova. The tool I made was crap, but it gave me insight into how the game worked. So, drawing upon nostalgia and old knowledge of the game I decided that rebuilding the game engine, with modern technologies, was the only solution.

This presented a slight problem…

ResourceForks & HFS+

With the release of Mac OS X almost 17 years ago, Apple declared that the ResourceFork was “legacy” technology and should no longer be used for new projects. However the API’s were still present for old programs. Great!

With the migration to 64-bit, Apple declared that the Carbon framework was now deprecated and end-of-life. The Carbon framework is where the ResourceFork functionality lives/lived. Uh-oh.

As Apple continued marching onwards the ResourceFork became less and less reliable. EV Nova received an update to deal with this. They migrated all the data files over to the ndat format. ndat is nothing special. It’s simply the ResourceFork data in a simple DataFork file. However the contents are a proprietary Apple format. No problem, the old Carbon functions could parse it. They maybe deprecated but they weren’t going anywhere anytime soon!

But it poses a problem. The old Carbon frameworks will be thrown out at some point. And when they are EV Nova will stop working. Those frameworks are the only part of Apple’s systems that can read the old ResourceFork format.

Now it’s true that the windows version doesn’t use the ndat format. It uses a custom rez format. So in theory that functionality could be moved over to the Mac version. But that would be a huge undertaking, and given the state of Ambrosia Software these days, probably not very likely.

So there is only one future proof solution… make a custom implementation of the parser for the ndat format (and one for rez at the same time).

QuickDraw

Awesome! We’re going to be up and running in no time! We’ll have all the old resources extracted out and we’ll be able to start building a game engine.

Uh-oh! Turns out resources such as PICT, cicn, RLË, ppät, etc, were all encoded/decoded via QuickDraw functionality, and that has long since gone the way of the dodo on macOS. Though curiously Preview can still render a PICT? Either way there is no publicly available way to handle those resources.

So that means a need to reverse engineer those as well… this may take a while.

A slight hint of a silver lining

There is some good that will come out of this project.

It will hopefully result in a modern version of the game engine that will continue to work for years to come.
Old total conversions such as Polycon EV, and even the Escape Velocity, and Escape Velocity Override conversions will work in this new engine with modification to the files.
It’s modern and will have custom parsers included so it wouldn’t be too much a stretch to get it running on iOS (though controls maybe an issue.)

I plan on using the blog to keep people who are interested up to date with the progress of work and to document some of the technical aspects as well.

Hopefully this will be of interest to some people…