Plugin plans in Kestrel

In my last post I talked about how the EV: Override revival project is now a thing and how the engine is being called Kestrel. In that post I also mentioned about the plans for plugin development and how maintaining backwards compatibility is an important factor. At the same time it will be important to expand the capabilities and features afforded to plugin developers.

This post will explain how this will be done and my current plans on accomplishing it.

ResourceForks

I have talked in the past about ResourceForks and how they are a central aspect of how the old Escape Velocity games worked. Everything was stored in ResourceForks, and that allowed the engine to quickly located the required resources, and for plugin developers to quickly identify, modify and replace existing resources in the game.

This functionality will be included into kestrel and will allow it to load all of the old legacy data files and plugins. Plugins using this format will be constrained to the same limitations as they were back on the original engines, as some of the constraints actually came from the format itself.

Newer content should be able to make use of a custom extension on ResourceForks.

Extended ResourceForks

So this is a purely custom thing, based upon the ResourceFork format. The format will allow for larger offset numbers and Resource IDs, thus removing limitations (for all practical purposes) of the format. These values will likely become 64-bit values, rather than 16-bit values.

This will be the extent of the change to the ResourceFork format.

Distinguishing between the two formats

So how will the engine distinguish between the formats, and what will it mean for plugin developers?

Well it won’t really mean anything for plugin developers, or even much for the engine itself. Most of the heavy work will be handled by libResourceFork which is part of the Diamond Project. That will handle loading from both formats and then providing back a common set of data structures and interfaces.

Internally though there will be a small addition to the header of the ResourceFork to denote that it is in the extended format.

Let’s take a look at the header (preamble) of the original ResourceFork.

Standard ResourceFork Preamble

This provides information about the layout of the ResourceFork. We can visualise this slightly with the following diagram.

The basic layout / structure of a ResourceFork

The two offset values represent the position within the file for that particular structure. The “data_offset” being where the actual data for all of the resources is located, and the “map_offset” being where all of the resource meta data is located (including types, ids, names, flags, etc). The corresponding sizes simply state how much data of each there is.

Note: Technically the resource map could be located before the resource data, but everything I’ve seen has this ordering.

So let’s take a look at the proposed preamble for the extended format.

Extended ResourceFork Preamble

There are two big changes. Firstly is an extra field at the start of the structure, encoding information about the format.

Due to the extremely unlikely scenario of both the “data_offset” and “map_offset” having a combined value of 1, this is the value we expect “format_version” to return. Anything else causes the file to be treated as a standard ResourceFork.

The second change is the use of 64-bit values, effectively making the capacity of the ResourceFork infinite (unless you have access to 18.5EB of storage and can fill it.)

So why/how does this “format_version” field trick work?

Let’s take a look at the initial 20 bytes of a standard ResourceFork.

A standard ResourceFork preamble

Each grouping of 8 digits represents a single 32-bit value (each pair of digits is a single byte). Using the above structure for the standard preamble we can see that the “data_offset” field will have a value of “00000100”, which translates to 256. We can also see that the “map_offset” field will have a value of “00007520” which translates to 29,984.

Now if we apply the extended preamble format we need to take 16 digits in order to have a 64-bit value. This will give our “format_version” field a value of “000001000007520” which translates to 1,099,511,657,760, which is quite a bit bigger than 1!

Because the offsets are from the beginning of the file, we can be certain that neither offset is going to be 0, and any value greater than 0 in “data_offset” will cause “format_version” to have a value in excess of 4,294,967,296.

So now that we have a method of distinguishing between standard and extended ResourceForks, let’s take a look at the resource map, and limitations that it imposes.

A diagram illustrating the layout of the resource map

This whole structure represents all of the meta data in the ResourceFork. Resource types, resource IDs, sizes, flags, resource names, etc. It is this structure that causes us most of our headaches and limitations.

The data structure representing the “resource map layout”

This data structure is only using 16-bit fields! Given that both offsets included are from the beginning of the resource map, this means we have a limitation on the number of resources that can be included in the ResourceFork. Obviously we can split the resources into multiple files to get around this issue but that is not always a nice or elegant solution (plus I’m not totally certain on what the internal limitations of the Resource Manager were, so there may be further limitations there!)

So let’s work out, best case scenario, how many resources we could house in a single ResourceFork. We’ll assume that we have just a single resource type for this. Let’s take a look at the structure of the type list.

The resource type list contains a count of how types are included in the file, followed by a list of type definitions, which in turn point to a list of resource definitions.

This structure may seem complex, but in reality it is not that bad. The type count field at the start is a 16 bit value, which means a total of 65,536 types can be included in a single ResourceFork. For all practical purposes this would never be reached.

Each type entry is 8 bytes long. 4 bytes denoting a type code such as ‘PICT’, 2 bytes for the offset to the first resource of that type and then a further 2 bytes representing how resources of that type are included.

The resource entry is 12 bytes long. 2 bytes for the resource ID, 2 bytes for the offset to the resource name, 1 byte for the resource flags, 3 bytes for the offset to the resource data and then 4 ignored bytes.

Note: These bytes are not actually ignored and used by the system for an unknown purpose, likely storing a reference value.

So we started with a total of 65,536 bytes of space. Immediately we can take away 6 bytes for the resource map structure, followed by a further 2 bytes for the resource type count, and then finally another 8 bytes for our type definition.

This gives us a total 65,520 bytes of space to keep track of our resources, which when divided between resource definitions allows us to store a maximum of 5,460 resources in a single ResourceFork! This means that a game such as EV Nova requires its data to be split across multiple files. Assuming those resources do not need more than 16MB of data (due to the 3-byte data offsets the format uses.)

In reality memory constraints on earlier systems will have been a much more limiting factor.

How the Extended ResourceFork will change things.

I already mentioned the changes to the preamble structure at the start of the ResourceFork. This is purely to help differentiate the two formats. The next change will be to make all of the 16-bit (2 byte) values in to 64-bit values.

In addition to that all resource names will be UTF-8 encoded rather than Mac Roman Encoded, allowing for modern representation of text in the resource names.

However this new format will need a proper specific defining at some point, but keen plugin developers will have noticed an issue with this proposal. It will break existing plugin editors!

The Kestrel Plugin Development Kit (KPDK)

This is a tangental project to Kestrel and will be extremely barebones, likely requiring the community to make it nice (kind of like how Nova Tools and Mission Computer made EV Nova plugin development nice).

So what is the plan with this? What is it?

Many people have expressed concern over the use of resource forks, and rightly so. They are an old format, and if backwards compatibility was not a concern they would not be getting used. But this does not mean plugin developers should be subjected to them. For this reason the KPDK will be compiler of sorts that compiles simple text based definitions into plugins compatible with Kestrel or EV Nova.

Here is a simple draft definition for a plugin.

; Define the planet Earth
StellarObject {
    New(id = #128, name = "Earth") {
        PlanetType = 0
        Government = #128
        ; Other fields for the stellar object.
    }
}

; Define a planet description for Earth
Description {
    New(id = StellarObject("Earth"), name = "Earth Landing Description") {
        Body = "Welcome to Earth. The home of Humanity."
        LandingPicture = PNG(/path/to/image/file.png)
    }
}

The compiler would know how to parse this and how to build the appropriate resources. It would be able synthesise the appropriate resources and reference resources with in definitions, allowing a plugin developer to not worry about remembering specific id ranges or how to link things.

There will be more information on this KPDK later on once it has been fleshed out more, and once Kestrel is actually loading some data files.

But I do plan to have compiler be able to target both Kestrel and EV Nova, as well as provide warnings on issues in the definitions being compiled and to be able to decompile a Kestrel plugin, or EV Nova plugin in to this definition script.

It should also mean that the tooling and features to plugin developers can evolve and expand over time without being tightly bound and dependant on the engine.

Wrap up

There was a lot of information in this post. I hope you enjoyed reading it, and it gives some insight into the direction I hope to go with regards to plugins.

6 thoughts on “Plugin plans in Kestrel

  1. As I was reading the post, I was definitely concerned about the reliance on resource forks, until seeing at the end that you were proposing a more friendly (and GitHub-diffable) syntax for modern plugin developers. Very pleased to see that! I wouldn’t want to have to fire up ResEdit to make plugins!

    However, I’m a bit confused about the syntax you’ve proposed. Wouldn’t it be simpler, and lower the learning curve, to just make it a standardized markup language like JSON or YAML? So your example would look something like this:

    {
    “StellarObjects”: [
    {
    “id”: 128,
    “name”: “Earth”,
    “PlanetType”: 0,
    “Government”: 128,
    “Landing Description”: “Welcome to Earth. The Home of Humanity.”
    “Landing Picture”: “/path/to/image/file.png”
    }
    ]
    }

    Note that I combined the Description in the StellarObjects. Having the Description be a separate object seems like a limitation of the ResourceFork format, and I think it’s far more logical to put it together since there is no such limitation with modern stuff.

    Like

    1. The ResourceFork format is being used purely to allow for backwards compatibility, and no, I would not expect anyone to use ResEdit, hence why I’m setting up the KPDK.

      There has actually been some iteration on it since what I posted, but I’m waiting to have something a bit more fleshed out before posting an update.

      One of the things I’ve since tried to emphasise with it is that this will be a community led project. The foundation that I make for KPDK will be to serve the absolute most basic needs of the engine and something that suits _my_ needs. I personally don’t like working with JSON and/or YAML, and so have opted to make a DSL for it.

      This foundation could very easily be expanded by others to add support for JSON, YAML, XML, etc. Of course I need to get the foundation sorted out first and the specification for it. However I’ve been focusing on the actual Kestrel engine more recently.

      Like

      1. I guess the bigger question is: what is the benefit of storing new data in this new Extended ResourceFork as opposed to having new data in a modern format and having a conversion process to go from the old (regular) ResourceFork format to the new one? I’m not sure what is gained by preserving a 25 year old data structure when there are better options today.

        I’m surprised to hear that you don’t like working with JSON and YAML; I find it very simple to use and by having an already established syntax, there is zero learning curve for someone already familiar with the format than for a DSL, for which nobody will be familiar with it. Also, I’m a bit unsure of what this looks like as a community led process if the source code will be closed source before the initial release. How can the community help make these decisions if they are all behind the curtain, so to speak? I’d love to add support for these sorts of languages, but I’m not sure how.

        Like

      2. Fundamentally there is none, but I could reverse the question.

        Binary formats are easier to work with, they have less overhead and can represent basically anything you throw at them or define. For this reason alone the engine will use a binary format. Are there better options for binary containers today? Certainly, but it just adds a burden on to the end user for converting older plugins and total conversions. It’s pointless the engine reading those older files and converting the newer ones, only to then read the newer ones. Why not just read the older one? What difference does it make?

        People often are surprised. My day job is as an iOS App Developer. I work with JSON and YAML daily. Either as data being communicated with a server or as configuration files for CI/CD systems. I honestly don’t know why people like them so much. I prefer YAML over JSON, but it’s still not a “language” that I like. They just look messy and the lack of concrete type information is annoying.

        So KPDK will be community led, not Kestrel itself. Kestrel as I mentioned in my previous comment is going to be open sourced eventually, but the chances are the license will be mildly restrictive (i.e. none commercial use, etc). KPDK will be community led as in, I don’t really care what direction it takes. The actual base project that I’m sorting out is more of a specification and a starting point. Seriously a blank repo is not a good starting point 😊. I don’t have any code written for it currently. I have said a number of times that if someone wants to start it then they are free to do so.

        As I explained before anything I add into KPDK will be purely for my benefit to make certain things for me easier. That’s what the initial specification is for, so people know what I’m expecting out of it. Where it goes beyond there is anyone’s guess.

        Like

  2. Is this Extended ResourceFork going to be in the actual ResourceFork of the file, or is it more of a simulated ResourceFork? I ask because Apple hasn’t really used the ResourceFork since Mac OS X debuted nearly 20 years ago, and given that you’ve noted in other posts that there isn’t a standard API for accessing it, I’m concerned that at some point Apple may drop support for it entirely, which would undo any work you’ve built off of it. (Also, I’m guessing Windows can’t use a ResourceFork).

    Like

    1. This is a “simulated” ResourceFork, and due to it being an entirely separate implementation from Apple’s own, it will work on systems other than macOS.

      The ResourceFork is actually a feature of the file system, not the OS. The binary data and format is a feature of the OS (or more specifically the old Resource Manager implementation), but the actual _ResourceFork_ is a file system feature. Basically in some file systems, files can be split into multiple “forks”. macOS has historically had two forks, the Data and Resource Forks. Under newer versions of macOS (using the Apple File System) the ResourceFork data has become an Extended File Attribute, ‘xattr’, with the key “com.apple.ResourceFork”)

      Sorry for the slightly confused mess above… I might make another small post about this.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s