Breaking the 64k barrier

Juan Luis
Member
Posts: 10
Joined: Wed May 20, 2020 7:30 pm

Breaking the 64k barrier

Post by Juan Luis »

My questions are:

- Is it possible to create programs that manages more than 64KBytes of data with MSX linking for MSX-DOS?
- If the answer is yes, is it possible for cartridges (ROM format)?
- Is it possible to create programs with more than 64KBytes of code?
- If the answer is yes, how?

I'm trying to compile an MSX-DOS program allocating __far pointers, but I have got error and warning messages.

Code: Select all

sccz80:"main.c" L:111 Warning:Implicit definition of function 'malloc_far' it will return an int. Prototype it explicitly if this is not what you want. [-Wimplicit-function-definition]
sccz80:"main.c" L:116 Warning:Implicit definition of function 'free_far' it will return an int. Prototype it explicitly if this is not what you want. [-Wimplicit-function-definition]
Error at file 'main.c' line 767: symbol '_malloc_far' not defined
Error at file 'main.c' line 821: symbol 'lp_plong' not defined
Error at file 'main.c' line 831: symbol '_free_far' not defined
Errors in source file D:\Documentos\blueMSXv282full\Tools\z88dk\lib\config\..\..\\lib\target\msx\classic\msx_crt0.asm:
Error at file 'main.c' line 767: symbol '_malloc_far' not defined
                   ^ ---- (null)Error at file 'main.c' line 821: symbol 'lp_plong' not defined
                   ^ ---- (null)Error at file 'main.c' line 831: symbol '_free_far' not defined
                   ^ ---- (null)
I'm compiling with these options:

Code: Select all

zcc +msx -c transform.c
zcc +msx -lm -o prueba.com -subtype=msxdos transform.o main.c
I got same errors trying these options:

Code: Select all

zcc +msx -DAMALLOC -c transform.c
zcc +msx -DAMALLOC -lm -o prueba.com -subtype=msxdos transform.o main.c
I have also tried the following:

Code: Select all

zcc +msx -DFARDATA -c transform.c
zcc +msx -DFARDATA -lm -o prueba.com -subtype=msxdos transform.o main.c
With -DFARDATA the error messages were a little bit different:

Code: Select all

Error at file 'main.c' line 767: symbol 'malloc_far' not defined
Error at file 'main.c' line 820: symbol 'lp_plong' not defined
Error at file 'main.c' line 830: symbol 'free_far' not defined
Errors in source file D:\Documentos\blueMSXv282full\Tools\z88dk\lib\config\..\..\\lib\target\msx\classic\msx_crt0.asm:
Error at file 'main.c' line 767: symbol 'malloc_far' not defined
                   ^ ---- (null)Error at file 'main.c' line 820: symbol 'lp_plong' not defined
                   ^ ---- (null)Error at file 'main.c' line 830: symbol 'free_far' not defined
                   ^ ---- (null)
Must I link to any specific library for far pointers?
Which one?

Thanks in advance.
Juan Luis
Member
Posts: 10
Joined: Wed May 20, 2020 7:30 pm

Post by Juan Luis »

One more thing. Can somebody explain how to use Trampoline calls? There isn't documentation about it.
User avatar
dom
Well known member
Posts: 2072
Joined: Sun Jul 15, 2007 10:01 pm

Post by dom »

There's no out-of-the-box support for supporting more than 64k for anything apart from the z88 (__far pointers) and __banked calls for the Gameboy.

__far pointers as used on the z88 are really only used for data - the allocation model on the z88 is that you ask for OS for some memory and it can provide you with a pointer that's anywhere in the 4MB address space. Applications on the z88 can't be more than 48kb in size. The workaround for this was to create "package" applications that register with a rst so that they can be accessed.

__banked calls for the Gameboy, is probably best explained by an example:

Code: Select all

extern void scall() __banked;


void func() {
        scall();
}
This will generate the following code:

Code: Select all

._func
        call    banked_call
        defq    _scall
        ret
The function banked_call is implemented by the platform library and is responsible for switching banks. The parameter following is calculated by z80asm - take a look at lib/target/gb/classic/gb_crt0.asm to see how functions can have 32 bit addresses.

For a more structured call there's__z88dk_shortcall(RSTNUMBER, CALLNUMBER).

For the code:

Code: Select all

extern int scall(long x, int y) __z88dk_shortcall(8, 200);
extern int scall2(long x, int y) __z88dk_shortcall(8, 2000);

int func()
{
   return scall(1L, 2);
}

int func2()
{
   return scall2(1L, 2);
}
The following is generated:

Code: Select all

._func
        ld      hl,1    ;const
        ld      de,0
        push    de
        push    hl
        ld      hl,2    ;const
        push    hl
        rst     8
        defb    200
        pop     bc
        pop     bc
        pop     bc
        ret

._func2
        ld      hl,1    ;const
        ld      de,0
        push    de
        push    hl
        ld      hl,2    ;const
        push    hl
        rst     8
        defw    2000
        pop     bc
        pop     bc
        pop     bc
        ret
Which one you use is up to you, __banked is definitely more adhoc and probably more suited to standalone applications and __z88dk_shortcall more suited to calling a library of routines.

All this is great, however the banking code may well adjust the stack pointer such that parameters aren't at the offset that the (banked) function expects them to be, so you can annotate the banked function with __z88dk_params_offset(OFFSET) where OFFSET is an integer that is the number of bytes that the stack has been adjusted by.

I think for your use-case then using __banked is probably the best to use - as I mentioned there's nothing provided at the moment, but it should just be a case of providing the banked_call function and adding the extra sections and orgs to the CRT.
User avatar
dom
Well known member
Posts: 2072
Joined: Sun Jul 15, 2007 10:01 pm

Post by dom »

As a follow up question, which mapper is the most commonly used these days/what is it the page size? I can probably setup an example project and the required stubs if you let me know.
User avatar
dom
Well known member
Posts: 2072
Joined: Sun Jul 15, 2007 10:01 pm

Post by dom »

It looks like I completely misread the question - you were asking about msxdos, however I've added the infrastructure to create an MSX rom that's bigger than 64k.

z88dk/examples/banking shows inter bank calling on the MSX using the Konami mapper. In this model 0x4000-0x7fff is a common area that remains paged in - the intention is that C library stays in this area so it can be accessed from the paged in code/data which occupies the memory 0x8000 -> 0xbfff

For MSXDOS you could do something similar except with RAM paging rather than paging.
Timmy
Well known member
Posts: 392
Joined: Sat Mar 10, 2012 4:18 pm

Post by Timmy »

I was going to add some stuff in to make a Konami mapped ROM one day. (I was very busy lately and I even missed MSXDev.) I think I had my notes somewhere to add more data to a 48k rom.

But if you are already adding something in then I don't have to do it.
User avatar
dom
Well known member
Posts: 2072
Joined: Sun Jul 15, 2007 10:01 pm

Post by dom »

It turns out that it was fairly simple in the end - most of the bits were there, it just needed to be put together. There's an example here: https://github.com/z88dk/z88dk/tree/mas ... les/banked which should work on both the MSX and the Gameboy. It would probably be useful to add support for the Spectrum as well.

The feature should work with both compilers which is an added bonus (I've only tested with sccz80 though), the caveat is that sdcc only changes the code section with #pragma bank XX , whereas sccz80 also changes the rodata section.

Having to specify the argument offset for the functions isn't great - if banked_call (https://github.com/z88dk/z88dk/blob/mas ... om.asm#L74) was rewritten it could probably be removed.
Juan Luis
Member
Posts: 10
Joined: Wed May 20, 2020 7:30 pm

Post by Juan Luis »

dom wrote:It turns out that it was fairly simple in the end - most of the bits were there, it just needed to be put together. There's an example here: https://github.com/z88dk/z88dk/tree/mas ... les/banked which should work on both the MSX and the Gameboy. It would probably be useful to add support for the Spectrum as well.

The feature should work with both compilers which is an added bonus (I've only tested with sccz80 though), the caveat is that sdcc only changes the code section with #pragma bank XX , whereas sccz80 also changes the rodata section.

Having to specify the argument offset for the functions isn't great - if banked_call (https://github.com/z88dk/z88dk/blob/mas ... om.asm#L74) was rewritten it could probably be removed.
Thanks for adding the example in github repository. I'm taking a look on the code.
Juan Luis
Member
Posts: 10
Joined: Wed May 20, 2020 7:30 pm

Post by Juan Luis »

I have just replaced rom.asm with the new version in github and I was able to compile the example and it works fine :-)

After reading rom.asm source code, I guess I have 32 banks available for my program (really 30 because banks 00 and 01 are not defined). Doesn't it?

I have a doubt. I thought Konami SCC mappers only supported 8KBytes pages according to this web page about MSX ROM mappers.

http://bifi.msxnet.org/msxnet/tech/megaroms#konami16

However, the source code shows 16 KBytes pages. There is a Konami SCC 16KBytes mapper but I don't have documentation about it. Is rom.asm file using this mapper?

I'm interested on ASCII 16KBytes mapper because this mapper supports 255 pages of 16KBytes with up to 4MBytes of code/data.

Is it possible to have an ASCII 16K version replacing the MAPPER_ADDRESS_8000 defc instructions and adding more bank definitions up to bank number 255?
Timmy
Well known member
Posts: 392
Joined: Sat Mar 10, 2012 4:18 pm

Post by Timmy »

I haven't seen any of it but this is still a good code for *a* mapper. If it works then it's fine.

Yes, there are a lot of miniscule details to make an SCC compatible ROM and I'll probably have to do it myself anyway, but not today. (Probably also because it's now almost sunrise and I'm still typing.)

I have no plans for an ASCII 16K version though. It would take more time to implement than a Konami mapper, and it would be less useful. Use a random 16K mapper instead, like the one on this page. It's good for what you need, and it's now 5AM for me.

This code is still very nice to use on the GameBoy and while I have no use of it right now (as in my next GB game), there might be a time soon that I have to use that capability, so I suggest you keep the code as is.

My plans for this year is still a MSX game (yes, a 32K max game), a GB game and a ZX game. I might be able to get a good Konami mapper done in between but my own games go first. I've already lost a lot of time the last couple of months, so I prefer getting my priorities fixed first.
User avatar
dom
Well known member
Posts: 2072
Joined: Sun Jul 15, 2007 10:01 pm

Post by dom »

I need to make a few tweaks since I've realised I've missed something to make life a bit easier, but the intention is that in software the following models can be created:

1. Standard 16/32k ROM - Timmy's original code
2. "Arbitrarily sized" ROMs with banking

For ease, the memory for model 2 is arranged into 16kb segments so we end up with:

0x4000 - 0x7fff = crt0, startup, z88dk library routines, never paged out
0x8000 - 0xbfff = paged memory
0xc000 - 0xffff = RAM

This means in code and layout we talk about 16k pages, and it's only in the mapper code that we translate those logical pages into physical pages. So you can see in the banked_call function that's committed we double the bank number and map in that page to 0x8000, add 1 and page into 0xa000 - so emulating a 16kb mapper with the 8kb mapper.

Supporting any mapper should be easy - I can add an option for the ASCII 16kb mapper to use the correct write addresses for it.

With respect to the number of pages, to be honest I just got bored and didn't add all the sections that could be available - I figured that around 512kb would be enough to start with!

The Gameboy code won't change, all that's shared is the toolchain infrastructure.
Juan Luis
Member
Posts: 10
Joined: Wed May 20, 2020 7:30 pm

Post by Juan Luis »

"With respect to the number of pages, to be honest I just got bored and didn't add all the sections that could be available - I figured that around 512kb would be enough to start with!"

:) Don't worry dom. I have seen the way of changing ROM pages and I believe I can modify the code by myself adding more pages and supporting ASCII16 ROM mapper.

This post was very interesting for me because I was able to learn some details of z88dk internal code.

Thanks dom and Timmy.

P.D: Timmy, I hope you can develop a great game.
User avatar
dom
Well known member
Posts: 2072
Joined: Sun Jul 15, 2007 10:01 pm

Post by dom »

I ended up being distracted by getting a floating point library working on the Gameboy so didn't get around to this yesterday.

I've made my tweaks, provided some options for a couple of other mappers and warned about "violating" the memory model in appmake. The current state of play is now on the wiki here: https://github.com/z88dk/z88dk/wiki/Pla ... garom-mode
Timmy
Well known member
Posts: 392
Joined: Sat Mar 10, 2012 4:18 pm

Post by Timmy »

The problem with making MegaROMs on the MSX is that it isn't really a technical problem. The biggest problem with creating a custom type MegaROM is that now you'd need support for it.

Unlike the Spectrum, which has bank switching baked into its hardware, the MSX ROMs has the memory banking soldered onto the ROMs itself. And unlike the GameBoy ROMs, the MegaROMs file format does not store its memory controller type anywhere.

GameBoy emulators solve this problem by looking at the cartridge header to find the Memory Controller Type, but on the MSX, the solution to detect its controller type, is to 1) use a hand crafted database of well-known games along with the controller, or 2) use a mapper detection heuristic, or 3) both.

As a result, if you want to make a MegaROM that is NOT from one of the existing formats, you'd have problems getting it working on many emulators, and on older versions of emulators.

I saw in the repository that you've added some code for ASCII 16 mapper. I think that is nice.

My suggestion is to use only the ASC 16 mapper for now, and drop the others, especially the hybrid "Konami 16 mapper".

For ASC 16, it's preferable to use "LD (0x7000),a" to do bank switching (and 0x6000 for the other bank, though in reality most developers will only swap the 0x7000 bank anyway). And while it's normal to call a function to do bank switching, the heuristics here is to use this code liberally.

For example, add these Magic Numbers somewhere: 0x32,0x00,0x70,0x32,0x00,0x70,0x32,0x00,0x70,0x32,0x00,0x70 ( yes, it's the same thing 4 times :P ) If possible, in an uncompressed part of your code/data. It does not need to be invoked, but is purely used as a detection header.

(PS. I still prefer a solution with 8k bank switching like the Konami SCC, because it's really way more flexible, but we can always do it later. For now we should get rid of those hybrid solutions first.)
User avatar
dom
Well known member
Posts: 2072
Joined: Sun Jul 15, 2007 10:01 pm

Post by dom »

Now I'm confused.

As far as I can tell I'm not doing anything that's hybrid, possibly unorthodox, but not hybrid.

All the mapping code respects the addresses defined on https://www.msx.org/wiki/MegaROM_Mappers so do match the heuristics used - I've checked Mame + Takeda source code - they simply detect the number of ld (XXXX),a for each defined mapper and assume it's the one with the highest counts.

The tip regarding including a sequence of them is good one - since data may well end inadvertently including those sequences.
Juan Luis
Member
Posts: 10
Joined: Wed May 20, 2020 7:30 pm

Post by Juan Luis »

Dom, thanks for adding ASCII16 MegaROM support. It is helpful for me.
Timmy
Well known member
Posts: 392
Joined: Sat Mar 10, 2012 4:18 pm

Post by Timmy »

If you feel like your generated code is going to pass the heuristics, then it's not my problem any more. I'm not going to dig into your code to see if it is.

I've been doing my own research on the heuristic functions too, otherwise I couldn't have give you all these information. (You'd be surprised how much I'd read on it and to condense it into just a few lines of text in my previous post.) It's a lot of time investing in something that I probably will never use, except for making a warning against unconventional ROM types. To make sure that the ROMs made with z88dk work in the real world.

And it's 4am again and instead of focusing on my own problems I'm just writing this. Feel free to ignore the warning.

Oh well. Good luck with it. I'm off to bed.
desertkun
New member
Posts: 4
Joined: Sat Dec 05, 2020 12:39 am

Re: Breaking the 64k barrier

Post by desertkun »

I've been looking at this wiki https://github.com/z88dk/z88dk/wiki/More-Than-64k

But my application is different: spectranet. I do not care for the data for the moment, and want to figure out "code".

Spectranet has 128k of extra ram, which are divided by 4k pages, and can be page in in 0x2000 ... 0x2FFF, 32 pages total. It has this system of modules, when any module function can be called via MODULECALL with module id in H and function number in L. A module can fit up to 4k and there can be 256 functions. Interesting thing is, module call pushes page onto the stack and when called function exits, it pops previous page off the stack, giving the caller a chance to resume. So page a can call page b and so on.

As far as I get it, __banked can work for me. But that still leaves need to manually track function banks and so on. 4k is going to be exhausted fast and off next page I go. I wonder if this can become part of language as well? So compiler would assign banks automatically, given user provides page call routine, a number of pages and size of each page.

E.g. z88dk would:

1. User supplies a page caller (say dynamic_banked_call) routine in general space, and it is used in linker
2. On link phase, assign each object file that is marked as dynamic bank space or something, a page number, or bank number, given the bank is 4k in size. Stack multiple object files together until we reach (or close to) 4k, so e.g. two object files of size 2k would share the same page.
3. Every function call of those object files would be generated as such: LD HL, local_offset; LD DE, bank_id; CALL dynamic_banked_call
6. A dynamic_banked_call would push current page onto the stack, switch the page to bank_id and call local_offset
7. On call return, pop the page off stack and switch to it

As far as I understand current design ideas, what's available on __banked isn't paged on stack, so the programmer has to keep track on which function leaves on what bank. By stacking object files into available banks and keeping bank number on the stack, we can restore banks and one banked function can call other banked function without the need to know on which bank it relies.

One more thing, and very important one: debugger; I want such code to give clean backtrace; So regardless of page caller mechanism, a calling convention would be, 2 bytes next to return address is a bank page and the debugger should switch briefly to that page to get frame pointer data at that point of time, etc.
Timmy
Well known member
Posts: 392
Joined: Sat Mar 10, 2012 4:18 pm

Re: Breaking the 64k barrier

Post by Timmy »

Just to make sure, Spectranet is for the Spectrum, right?

And that you are talking about an implementation for the Spectrum, or is it on the Spectranet hardware itself?
desertkun
New member
Posts: 4
Joined: Sat Dec 05, 2020 12:39 am

Re: Breaking the 64k barrier

Post by desertkun »

That is correct https://www.bytedelight.com/?page_id=3515

It is for the spectrum. The cartridge, spectranet, can page-in in the first 16k of memory, it could be ram or rom. 32 4k pages of ram and 32 4k pages of rom, so I am asking about ram. The program starts, deploys its code to ram pages and starts executing. I wonder if how we can attack the banking problem so compiler can assign banks automatically, and then stack the banks so one page can safely can call the other, even though they share the same memory block.
User avatar
dom
Well known member
Posts: 2072
Joined: Sun Jul 15, 2007 10:01 pm

Re: Breaking the 64k barrier

Post by dom »

Something that does this was mentioned here: https://www.z88dk.org/forum/viewtopic.p ... 266#p20266

Fuzix does something similar - Alan describes it on the rc2014 mailing list amongst other places.
nuc1e0n
Member
Posts: 49
Joined: Wed Jun 10, 2020 12:34 am

Re: Breaking the 64k barrier

Post by nuc1e0n »

> (malloc_far and free_far are undefined)

According to https://github.com/z88dk/z88dk/wiki/More-Than-64k, functions to manage a far heap need to be written by yourself. My understanding is these functions declarations merely act as hooks for the standard library.

> Can somebody explain how to use Trampoline calls? There isn't documentation about it.

The concept of a trampoline is just that, a concept. I believe z88dk provides several different mechanisms to create and use them, but leaves which approach to use up to you as the developer of your application.

In the abstract, a trampoline is a function that loads a function into ram that isn't currently in memory.
Once this function has been loaded into ram the trampoline then jumps to this newly loaded code.
So a trampoline lets you 'jump' further than you otherwise could ;) https://en.wikipedia.org/wiki/Trampoline_(computing)

The memory layout on MSX machines is rather inconsistent from one model to another. My understanding of the general concept of the MSX is that the capabilities of the machine would be made accessible and used via software API calls, for which manufacturers would connect up to their own specific hardware configurations.

This is why MSXDOS2 has a bios extension to control memory mapping. The hardware i/o ports behind the scenes can and will be different.

As has been said on this thread, z88dk has some support for creating MegaROMs for the MSX, but unfortunately there's no standard
MSX emulator format that describes the memory mapper used by any particular cartridge image, equivalent to the iNES format for NES games or the .crt format for commodore 64 cartridges. MSX cartridge images are merely a dump of the contents of their ROM.

For this reason, for my project I stick to using the MSXDOS2 bios extension for memory mapping on the MSX. Anything else was just too unpredictable for the software I wanted to write.

Using the MSXDOS2 memory mapping facilities, you can allocate or deallocate whole 16k blocks of RAM and switch them into the address space as and when you need them. Because they are RAM you can use them for both code and for modifiable data.

I use these blocks for read only code in my project, but you could implement a far heap (malloc_far and free_far) using the msxdos2 memory management functionality and use these 16k blocks for your own modifiable data.
User avatar
dom
Well known member
Posts: 2072
Joined: Sun Jul 15, 2007 10:01 pm

Re: Breaking the 64k barrier

Post by dom »

As has been said on this thread, z88dk has some support for creating MegaROMs for the MSX, but unfortunately there's no standard
MSX emulator format that describes the memory mapper used by any particular cartridge image, equivalent to the iNES format for NES games or the .crt format for commodore 64 cartridges. MSX cartridge images are merely a dump of the contents of their ROM.
Looking at the source code of the emulators, they determine the mapper type based on counting the number of writes to certain addresses which is why the crt0 attempts to bias using this bit of code: https://github.com/z88dk/z88dk/blob/mas ... m.asm#L101 - it seems to work

It's a bit of a theme I've noticed with MSX file formats: they're generally just raw files, eg .dsk files are actually just a raw sector dump in a particular order.
nuc1e0n
Member
Posts: 49
Joined: Wed Jun 10, 2020 12:34 am

Re: Breaking the 64k barrier

Post by nuc1e0n »

I think it's a shame that MSX emulator authors haven't come up with a file format to directly specify which mapper to use. But I suppose if we want to make new software we've got to work with what we have, not what we wish we had huh?
Timmy
Well known member
Posts: 392
Joined: Sat Mar 10, 2012 4:18 pm

Re: Breaking the 64k barrier

Post by Timmy »

nuc1e0n wrote: Tue Feb 08, 2022 12:25 pm I think it's a shame that MSX emulator authors haven't come up with a file format to directly specify which mapper to use. But I suppose if we want to make new software we've got to work with what we have, not what we wish we had huh?
A Mapper is Hardware. There are more or less a billion and one ways to make mappers and chips and wires and solders (basically it's like the many types of MSX machines but ROMs, being simpler, have even more variations.) If you think it's really easy to specify them all and write loaders for them, I'd definitely suggest you to start today!

This flexibility is also the reason that making up the another type op mapper -- like we do here -- is going to have emulator authors to add another entry and code into their emulator to just make sure your game will work. And they will have to do that every single time a new game/original mapper is made up, because people always end up doing what they wish they had. Or they'll just skip it.

The 32K ROM format I made earlier for z88dk, that should be working for all kinds of machines and emulators, because that's a tried and tested setup. As for MegaROMs, well, you've already built "the thing you wish you had" and not "what emulators already offer." :)

Anyway, nice rhetorical question, so I am not going to answer it. But, good question there.
Post Reply