Plugin plans in Kestrel

In my last post I talked about how the EV: Override revival project is now a thing and how the engine is being called Kestrel. In that post I also mentioned about the plans for plugin development and how maintaining backwards compatibility is an important factor. At the same time it will be important to expand the capabilities and features afforded to plugin developers.

This post will explain how this will be done and my current plans on accomplishing it.

ResourceForks

I have talked in the past about ResourceForks and how they are a central aspect of how the old Escape Velocity games worked. Everything was stored in ResourceForks, and that allowed the engine to quickly located the required resources, and for plugin developers to quickly identify, modify and replace existing resources in the game.

This functionality will be included into kestrel and will allow it to load all of the old legacy data files and plugins. Plugins using this format will be constrained to the same limitations as they were back on the original engines, as some of the constraints actually came from the format itself.

Newer content should be able to make use of a custom extension on ResourceForks.

Extended ResourceForks

So this is a purely custom thing, based upon the ResourceFork format. The format will allow for larger offset numbers and Resource IDs, thus removing limitations (for all practical purposes) of the format. These values will likely become 64-bit values, rather than 16-bit values.

This will be the extent of the change to the ResourceFork format.

Distinguishing between the two formats

So how will the engine distinguish between the formats, and what will it mean for plugin developers?

Well it won’t really mean anything for plugin developers, or even much for the engine itself. Most of the heavy work will be handled by libResourceFork which is part of the Diamond Project. That will handle loading from both formats and then providing back a common set of data structures and interfaces.

Internally though there will be a small addition to the header of the ResourceFork to denote that it is in the extended format.

Let’s take a look at the header (preamble) of the original ResourceFork.

This provides information about the layout of the ResourceFork. We can visualise this slightly with the following diagram.

The two offset values represent the position within the file for that particular structure. The “data_offset” being where the actual data for all of the resources is located, and the “map_offset” being where all of the resource meta data is located (including types, ids, names, flags, etc). The corresponding sizes simply state how much data of each there is.

Note: Technically the resource map could be located before the resource data, but everything I’ve seen has this ordering.

So let’s take a look at the proposed preamble for the extended format.

There are two big changes. Firstly is an extra field at the start of the structure, encoding information about the format.

Due to the extremely unlikely scenario of both the “data_offset” and “map_offset” having a combined value of 1, this is the value we expect “format_version” to return. Anything else causes the file to be treated as a standard ResourceFork.

The second change is the use of 64-bit values, effectively making the capacity of the ResourceFork infinite (unless you have access to 18.5EB of storage and can fill it.)

So why/how does this “format_version” field trick work?

Let’s take a look at the initial 20 bytes of a standard ResourceFork.

Each grouping of 8 digits represents a single 32-bit value (each pair of digits is a single byte). Using the above structure for the standard preamble we can see that the “data_offset” field will have a value of “00000100”, which translates to 256. We can also see that the “map_offset” field will have a value of “00007520” which translates to 29,984.

Now if we apply the extended preamble format we need to take 16 digits in order to have a 64-bit value. This will give our “format_version” field a value of “000001000007520” which translates to 1,099,511,657,760, which is quite a bit bigger than 1!

Because the offsets are from the beginning of the file, we can be certain that neither offset is going to be 0, and any value greater than 0 in “data_offset” will cause “format_version” to have a value in excess of 4,294,967,296.

So now that we have a method of distinguishing between standard and extended ResourceForks, let’s take a look at the resource map, and limitations that it imposes.

This whole structure represents all of the meta data in the ResourceFork. Resource types, resource IDs, sizes, flags, resource names, etc. It is this structure that causes us most of our headaches and limitations.

This data structure is only using 16-bit fields! Given that both offsets included are from the beginning of the resource map, this means we have a limitation on the number of resources that can be included in the ResourceFork. Obviously we can split the resources into multiple files to get around this issue but that is not always a nice or elegant solution (plus I’m not totally certain on what the internal limitations of the Resource Manager were, so there may be further limitations there!)

So let’s work out, best case scenario, how many resources we could house in a single ResourceFork. We’ll assume that we have just a single resource type for this. Let’s take a look at the structure of the type list.

This structure may seem complex, but in reality it is not that bad. The type count field at the start is a 16 bit value, which means a total of 65,536 types can be included in a single ResourceFork. For all practical purposes this would never be reached.

Each type entry is 8 bytes long. 4 bytes denoting a type code such as ‘PICT’, 2 bytes for the offset to the first resource of that type and then a further 2 bytes representing how resources of that type are included.

The resource entry is 12 bytes long. 2 bytes for the resource ID, 2 bytes for the offset to the resource name, 1 byte for the resource flags, 3 bytes for the offset to the resource data and then 4 ignored bytes.

Note: These bytes are not actually ignored and used by the system for an unknown purpose, likely storing a reference value.

So we started with a total of 65,536 bytes of space. Immediately we can take away 6 bytes for the resource map structure, followed by a further 2 bytes for the resource type count, and then finally another 8 bytes for our type definition.

This gives us a total 65,520 bytes of space to keep track of our resources, which when divided between resource definitions allows us to store a maximum of 5,460 resources in a single ResourceFork! This means that a game such as EV Nova requires its data to be split across multiple files. Assuming those resources do not need more than 16MB of data (due to the 3-byte data offsets the format uses.)

In reality memory constraints on earlier systems will have been a much more limiting factor.

How the Extended ResourceFork will change things.

I already mentioned the changes to the preamble structure at the start of the ResourceFork. This is purely to help differentiate the two formats. The next change will be to make all of the 16-bit (2 byte) values in to 64-bit values.

In addition to that all resource names will be UTF-8 encoded rather than Mac Roman Encoded, allowing for modern representation of text in the resource names.

However this new format will need a proper specific defining at some point, but keen plugin developers will have noticed an issue with this proposal. It will break existing plugin editors!

The Kestrel Plugin Development Kit (KPDK)

This is a tangental project to Kestrel and will be extremely barebones, likely requiring the community to make it nice (kind of like how Nova Tools and Mission Computer made EV Nova plugin development nice).

So what is the plan with this? What is it?

Many people have expressed concern over the use of resource forks, and rightly so. They are an old format, and if backwards compatibility was not a concern they would not be getting used. But this does not mean plugin developers should be subjected to them. For this reason the KPDK will be compiler of sorts that compiles simple text based definitions into plugins compatible with Kestrel or EV Nova.

Here is a simple draft definition for a plugin.

; Define the planet Earth
StellarObject {
    New(id = #128, name = "Earth") {
        PlanetType = 0
        Government = #128
        ; Other fields for the stellar object.
    }
}

; Define a planet description for Earth
Description {
    New(id = StellarObject("Earth"), name = "Earth Landing Description") {
        Body = "Welcome to Earth. The home of Humanity."
        LandingPicture = PNG(/path/to/image/file.png)
    }
}

The compiler would know how to parse this and how to build the appropriate resources. It would be able synthesise the appropriate resources and reference resources with in definitions, allowing a plugin developer to not worry about remembering specific id ranges or how to link things.

There will be more information on this KPDK later on once it has been fleshed out more, and once Kestrel is actually loading some data files.

But I do plan to have compiler be able to target both Kestrel and EV Nova, as well as provide warnings on issues in the definitions being compiled and to be able to decompile a Kestrel plugin, or EV Nova plugin in to this definition script.

It should also mean that the tooling and features to plugin developers can evolve and expand over time without being tightly bound and dependant on the engine.

Wrap up

There was a lot of information in this post. I hope you enjoyed reading it, and it gives some insight into the direction I hope to go with regards to plugins.