Breaking the 64k barrier

Post Reply
Juan Luis
Member
Posts: 10
Joined: Wed May 20, 2020 7:30 pm

Breaking the 64k barrier

Post by Juan Luis »

My questions are:

- Is it possible to create programs that manages more than 64KBytes of data with MSX linking for MSX-DOS?
- If the answer is yes, is it possible for cartridges (ROM format)?
- Is it possible to create programs with more than 64KBytes of code?
- If the answer is yes, how?

I'm trying to compile an MSX-DOS program allocating __far pointers, but I have got error and warning messages.

Code: Select all

sccz80:"main.c" L:111 Warning:Implicit definition of function 'malloc_far' it will return an int. Prototype it explicitly if this is not what you want. [-Wimplicit-function-definition]
sccz80:"main.c" L:116 Warning:Implicit definition of function 'free_far' it will return an int. Prototype it explicitly if this is not what you want. [-Wimplicit-function-definition]
Error at file 'main.c' line 767: symbol '_malloc_far' not defined
Error at file 'main.c' line 821: symbol 'lp_plong' not defined
Error at file 'main.c' line 831: symbol '_free_far' not defined
Errors in source file D:\Documentos\blueMSXv282full\Tools\z88dk\lib\config\..\..\\lib\target\msx\classic\msx_crt0.asm:
Error at file 'main.c' line 767: symbol '_malloc_far' not defined
                   ^ ---- (null)Error at file 'main.c' line 821: symbol 'lp_plong' not defined
                   ^ ---- (null)Error at file 'main.c' line 831: symbol '_free_far' not defined
                   ^ ---- (null)
I'm compiling with these options:

Code: Select all

zcc +msx -c transform.c
zcc +msx -lm -o prueba.com -subtype=msxdos transform.o main.c
I got same errors trying these options:

Code: Select all

zcc +msx -DAMALLOC -c transform.c
zcc +msx -DAMALLOC -lm -o prueba.com -subtype=msxdos transform.o main.c
I have also tried the following:

Code: Select all

zcc +msx -DFARDATA -c transform.c
zcc +msx -DFARDATA -lm -o prueba.com -subtype=msxdos transform.o main.c
With -DFARDATA the error messages were a little bit different:

Code: Select all

Error at file 'main.c' line 767: symbol 'malloc_far' not defined
Error at file 'main.c' line 820: symbol 'lp_plong' not defined
Error at file 'main.c' line 830: symbol 'free_far' not defined
Errors in source file D:\Documentos\blueMSXv282full\Tools\z88dk\lib\config\..\..\\lib\target\msx\classic\msx_crt0.asm:
Error at file 'main.c' line 767: symbol 'malloc_far' not defined
                   ^ ---- (null)Error at file 'main.c' line 820: symbol 'lp_plong' not defined
                   ^ ---- (null)Error at file 'main.c' line 830: symbol 'free_far' not defined
                   ^ ---- (null)
Must I link to any specific library for far pointers?
Which one?

Thanks in advance.
Juan Luis
Member
Posts: 10
Joined: Wed May 20, 2020 7:30 pm

Post by Juan Luis »

One more thing. Can somebody explain how to use Trampoline calls? There isn't documentation about it.
User avatar
dom
Well known member
Posts: 1194
Joined: Sun Jul 15, 2007 10:01 pm

Post by dom »

There's no out-of-the-box support for supporting more than 64k for anything apart from the z88 (__far pointers) and __banked calls for the Gameboy.

__far pointers as used on the z88 are really only used for data - the allocation model on the z88 is that you ask for OS for some memory and it can provide you with a pointer that's anywhere in the 4MB address space. Applications on the z88 can't be more than 48kb in size. The workaround for this was to create "package" applications that register with a rst so that they can be accessed.

__banked calls for the Gameboy, is probably best explained by an example:

Code: Select all

extern void scall() __banked;


void func() {
        scall();
}
This will generate the following code:

Code: Select all

._func
        call    banked_call
        defq    _scall
        ret
The function banked_call is implemented by the platform library and is responsible for switching banks. The parameter following is calculated by z80asm - take a look at lib/target/gb/classic/gb_crt0.asm to see how functions can have 32 bit addresses.

For a more structured call there's__z88dk_shortcall(RSTNUMBER, CALLNUMBER).

For the code:

Code: Select all

extern int scall(long x, int y) __z88dk_shortcall(8, 200);
extern int scall2(long x, int y) __z88dk_shortcall(8, 2000);

int func()
{
   return scall(1L, 2);
}

int func2()
{
   return scall2(1L, 2);
}
The following is generated:

Code: Select all

._func
        ld      hl,1    ;const
        ld      de,0
        push    de
        push    hl
        ld      hl,2    ;const
        push    hl
        rst     8
        defb    200
        pop     bc
        pop     bc
        pop     bc
        ret

._func2
        ld      hl,1    ;const
        ld      de,0
        push    de
        push    hl
        ld      hl,2    ;const
        push    hl
        rst     8
        defw    2000
        pop     bc
        pop     bc
        pop     bc
        ret
Which one you use is up to you, __banked is definitely more adhoc and probably more suited to standalone applications and __z88dk_shortcall more suited to calling a library of routines.

All this is great, however the banking code may well adjust the stack pointer such that parameters aren't at the offset that the (banked) function expects them to be, so you can annotate the banked function with __z88dk_params_offset(OFFSET) where OFFSET is an integer that is the number of bytes that the stack has been adjusted by.

I think for your use-case then using __banked is probably the best to use - as I mentioned there's nothing provided at the moment, but it should just be a case of providing the banked_call function and adding the extra sections and orgs to the CRT.
User avatar
dom
Well known member
Posts: 1194
Joined: Sun Jul 15, 2007 10:01 pm

Post by dom »

As a follow up question, which mapper is the most commonly used these days/what is it the page size? I can probably setup an example project and the required stubs if you let me know.
User avatar
dom
Well known member
Posts: 1194
Joined: Sun Jul 15, 2007 10:01 pm

Post by dom »

It looks like I completely misread the question - you were asking about msxdos, however I've added the infrastructure to create an MSX rom that's bigger than 64k.

z88dk/examples/banking shows inter bank calling on the MSX using the Konami mapper. In this model 0x4000-0x7fff is a common area that remains paged in - the intention is that C library stays in this area so it can be accessed from the paged in code/data which occupies the memory 0x8000 -> 0xbfff

For MSXDOS you could do something similar except with RAM paging rather than paging.
Timmy
Member
Posts: 136
Joined: Sat Mar 10, 2012 4:18 pm

Post by Timmy »

I was going to add some stuff in to make a Konami mapped ROM one day. (I was very busy lately and I even missed MSXDev.) I think I had my notes somewhere to add more data to a 48k rom.

But if you are already adding something in then I don't have to do it.
User avatar
dom
Well known member
Posts: 1194
Joined: Sun Jul 15, 2007 10:01 pm

Post by dom »

It turns out that it was fairly simple in the end - most of the bits were there, it just needed to be put together. There's an example here: https://github.com/z88dk/z88dk/tree/mas ... les/banked which should work on both the MSX and the Gameboy. It would probably be useful to add support for the Spectrum as well.

The feature should work with both compilers which is an added bonus (I've only tested with sccz80 though), the caveat is that sdcc only changes the code section with #pragma bank XX , whereas sccz80 also changes the rodata section.

Having to specify the argument offset for the functions isn't great - if banked_call (https://github.com/z88dk/z88dk/blob/mas ... om.asm#L74) was rewritten it could probably be removed.
Juan Luis
Member
Posts: 10
Joined: Wed May 20, 2020 7:30 pm

Post by Juan Luis »

dom wrote:It turns out that it was fairly simple in the end - most of the bits were there, it just needed to be put together. There's an example here: https://github.com/z88dk/z88dk/tree/mas ... les/banked which should work on both the MSX and the Gameboy. It would probably be useful to add support for the Spectrum as well.

The feature should work with both compilers which is an added bonus (I've only tested with sccz80 though), the caveat is that sdcc only changes the code section with #pragma bank XX , whereas sccz80 also changes the rodata section.

Having to specify the argument offset for the functions isn't great - if banked_call (https://github.com/z88dk/z88dk/blob/mas ... om.asm#L74) was rewritten it could probably be removed.
Thanks for adding the example in github repository. I'm taking a look on the code.
Juan Luis
Member
Posts: 10
Joined: Wed May 20, 2020 7:30 pm

Post by Juan Luis »

I have just replaced rom.asm with the new version in github and I was able to compile the example and it works fine :-)

After reading rom.asm source code, I guess I have 32 banks available for my program (really 30 because banks 00 and 01 are not defined). Doesn't it?

I have a doubt. I thought Konami SCC mappers only supported 8KBytes pages according to this web page about MSX ROM mappers.

http://bifi.msxnet.org/msxnet/tech/megaroms#konami16

However, the source code shows 16 KBytes pages. There is a Konami SCC 16KBytes mapper but I don't have documentation about it. Is rom.asm file using this mapper?

I'm interested on ASCII 16KBytes mapper because this mapper supports 255 pages of 16KBytes with up to 4MBytes of code/data.

Is it possible to have an ASCII 16K version replacing the MAPPER_ADDRESS_8000 defc instructions and adding more bank definitions up to bank number 255?
Timmy
Member
Posts: 136
Joined: Sat Mar 10, 2012 4:18 pm

Post by Timmy »

I haven't seen any of it but this is still a good code for *a* mapper. If it works then it's fine.

Yes, there are a lot of miniscule details to make an SCC compatible ROM and I'll probably have to do it myself anyway, but not today. (Probably also because it's now almost sunrise and I'm still typing.)

I have no plans for an ASCII 16K version though. It would take more time to implement than a Konami mapper, and it would be less useful. Use a random 16K mapper instead, like the one on this page. It's good for what you need, and it's now 5AM for me.

This code is still very nice to use on the GameBoy and while I have no use of it right now (as in my next GB game), there might be a time soon that I have to use that capability, so I suggest you keep the code as is.

My plans for this year is still a MSX game (yes, a 32K max game), a GB game and a ZX game. I might be able to get a good Konami mapper done in between but my own games go first. I've already lost a lot of time the last couple of months, so I prefer getting my priorities fixed first.
User avatar
dom
Well known member
Posts: 1194
Joined: Sun Jul 15, 2007 10:01 pm

Post by dom »

I need to make a few tweaks since I've realised I've missed something to make life a bit easier, but the intention is that in software the following models can be created:

1. Standard 16/32k ROM - Timmy's original code
2. "Arbitrarily sized" ROMs with banking

For ease, the memory for model 2 is arranged into 16kb segments so we end up with:

0x4000 - 0x7fff = crt0, startup, z88dk library routines, never paged out
0x8000 - 0xbfff = paged memory
0xc000 - 0xffff = RAM

This means in code and layout we talk about 16k pages, and it's only in the mapper code that we translate those logical pages into physical pages. So you can see in the banked_call function that's committed we double the bank number and map in that page to 0x8000, add 1 and page into 0xa000 - so emulating a 16kb mapper with the 8kb mapper.

Supporting any mapper should be easy - I can add an option for the ASCII 16kb mapper to use the correct write addresses for it.

With respect to the number of pages, to be honest I just got bored and didn't add all the sections that could be available - I figured that around 512kb would be enough to start with!

The Gameboy code won't change, all that's shared is the toolchain infrastructure.
Juan Luis
Member
Posts: 10
Joined: Wed May 20, 2020 7:30 pm

Post by Juan Luis »

"With respect to the number of pages, to be honest I just got bored and didn't add all the sections that could be available - I figured that around 512kb would be enough to start with!"

:) Don't worry dom. I have seen the way of changing ROM pages and I believe I can modify the code by myself adding more pages and supporting ASCII16 ROM mapper.

This post was very interesting for me because I was able to learn some details of z88dk internal code.

Thanks dom and Timmy.

P.D: Timmy, I hope you can develop a great game.
User avatar
dom
Well known member
Posts: 1194
Joined: Sun Jul 15, 2007 10:01 pm

Post by dom »

I ended up being distracted by getting a floating point library working on the Gameboy so didn't get around to this yesterday.

I've made my tweaks, provided some options for a couple of other mappers and warned about "violating" the memory model in appmake. The current state of play is now on the wiki here: https://github.com/z88dk/z88dk/wiki/Pla ... garom-mode
Timmy
Member
Posts: 136
Joined: Sat Mar 10, 2012 4:18 pm

Post by Timmy »

The problem with making MegaROMs on the MSX is that it isn't really a technical problem. The biggest problem with creating a custom type MegaROM is that now you'd need support for it.

Unlike the Spectrum, which has bank switching baked into its hardware, the MSX ROMs has the memory banking soldered onto the ROMs itself. And unlike the GameBoy ROMs, the MegaROMs file format does not store its memory controller type anywhere.

GameBoy emulators solve this problem by looking at the cartridge header to find the Memory Controller Type, but on the MSX, the solution to detect its controller type, is to 1) use a hand crafted database of well-known games along with the controller, or 2) use a mapper detection heuristic, or 3) both.

As a result, if you want to make a MegaROM that is NOT from one of the existing formats, you'd have problems getting it working on many emulators, and on older versions of emulators.

I saw in the repository that you've added some code for ASCII 16 mapper. I think that is nice.

My suggestion is to use only the ASC 16 mapper for now, and drop the others, especially the hybrid "Konami 16 mapper".

For ASC 16, it's preferable to use "LD (0x7000),a" to do bank switching (and 0x6000 for the other bank, though in reality most developers will only swap the 0x7000 bank anyway). And while it's normal to call a function to do bank switching, the heuristics here is to use this code liberally.

For example, add these Magic Numbers somewhere: 0x32,0x00,0x70,0x32,0x00,0x70,0x32,0x00,0x70,0x32,0x00,0x70 ( yes, it's the same thing 4 times :P ) If possible, in an uncompressed part of your code/data. It does not need to be invoked, but is purely used as a detection header.

(PS. I still prefer a solution with 8k bank switching like the Konami SCC, because it's really way more flexible, but we can always do it later. For now we should get rid of those hybrid solutions first.)
User avatar
dom
Well known member
Posts: 1194
Joined: Sun Jul 15, 2007 10:01 pm

Post by dom »

Now I'm confused.

As far as I can tell I'm not doing anything that's hybrid, possibly unorthodox, but not hybrid.

All the mapping code respects the addresses defined on https://www.msx.org/wiki/MegaROM_Mappers so do match the heuristics used - I've checked Mame + Takeda source code - they simply detect the number of ld (XXXX),a for each defined mapper and assume it's the one with the highest counts.

The tip regarding including a sequence of them is good one - since data may well end inadvertently including those sequences.
Juan Luis
Member
Posts: 10
Joined: Wed May 20, 2020 7:30 pm

Post by Juan Luis »

Dom, thanks for adding ASCII16 MegaROM support. It is helpful for me.
Timmy
Member
Posts: 136
Joined: Sat Mar 10, 2012 4:18 pm

Post by Timmy »

If you feel like your generated code is going to pass the heuristics, then it's not my problem any more. I'm not going to dig into your code to see if it is.

I've been doing my own research on the heuristic functions too, otherwise I couldn't have give you all these information. (You'd be surprised how much I'd read on it and to condense it into just a few lines of text in my previous post.) It's a lot of time investing in something that I probably will never use, except for making a warning against unconventional ROM types. To make sure that the ROMs made with z88dk work in the real world.

And it's 4am again and instead of focusing on my own problems I'm just writing this. Feel free to ignore the warning.

Oh well. Good luck with it. I'm off to bed.
Post Reply