Developing for MSX-DOS2 and >64kb

megatronuk
Member
Posts: 34
Joined: Fri Jan 28, 2022 11:39 pm

Re: Developing for MSX-DOS2 and >64kb

Post by megatronuk »

It would certainly be more advantageous if both MSX-DOS2 and DOS1 users had the ability to use banked function/data calls (limited to machines with mapper support, obviously). I see that both 'MU.COM' and 'MSR.COM' appear to do the same thing - add DOS2/MSX2 compatible mapper calls on machines with the hardware support to do so, but without the need for DOS2.

It's not as elegant (nor automatic) as the bank pragma approach documented in the examples, but I can certainly see the advantages over the relatively heavy DOS2 implementation.

That could be a significant improvement, since we know that for all the advantages that DOS2 brings, it has big memory requirement over DOS1....

Out of interest, how do you build your 'prefix00.ovl' bank files?
User avatar
dom
Well known member
Posts: 2072
Joined: Sun Jul 15, 2007 10:01 pm

Re: Developing for MSX-DOS2 and >64kb

Post by dom »

Thanks for all the reports, I think I've now fixed some issues:

1. The disc format generated by -create-app now seems to work.
2. Only 4 banks were being loaded because I stopped the loop early for debug purposes
3. I've added log messages and abort the binary when the banks aren't successfully loaded

In terms of OpenMSX hardware, I was using the OpenMSX boosted MSXTurboR since it seemed to have the mapper in the right place. The fact that it doesn't work on so many machines means that we need to be smarter about selecting the mapper and fail back from the primary to another mapper?

Regarding MSX-DOS1 support, it looks like MU provides the same EXTBIOS entry points so the code in z88dk would be applicable there - it's also broadly the same as the code from nuc1e0n
User avatar
dom
Well known member
Posts: 2072
Joined: Sun Jul 15, 2007 10:01 pm

Re: Developing for MSX-DOS2 and >64kb

Post by dom »

I forgot the screenshots:

Allocation failure to load on MSX2:
Screenshot 2022-02-06 at 17.00.17.png
Running on an MSX1 without a mapper/exbios:
Screenshot 2022-02-06 at 18.51.17.png
Successful:
Screenshot 2022-02-06 at 16.59.07.png
You do not have the required permissions to view the files attached to this post.
nuc1e0n
Member
Posts: 49
Joined: Wed Jun 10, 2020 12:34 am

Re: Developing for MSX-DOS2 and >64kb

Post by nuc1e0n »

I'm actually compiling them with my own fork of z88dk that has a new -cc-base command line option that I use to alter the stack offset of function local variables (here's a link to the only commit that's different to upstream z88dk: https://github.com/pjshumphreys/z88dk/c ... 1253a6126f)

Then I'm using a node.js script to orchestrate things (please don't judge. JavaScript is really good at manipulating big hash maps such as symbol tables). I use this script to slice up the assembly language output from z88dk into individual functions that I measure the binary sizes of and use a bin packing algorithm to pack the code into the minimum number of pages while still using information about which functions call which other ones to minimise page flipping.

The only things always kept in ram with my approach are the heap, stack, page flipping code and some of the C standard library code. All other code and constants (such as large arrays of constants and string literals) are paged in and out as necessary, using this memory map:

+---------------+
| 0x0000-0x00FF | cpm syscalls
+---------------+
| 0x0100-0x3FFF | paging code, cp/m or msx 2 file access routines, global variables and constants at bottom. extra heap memory at top
+---------------+
| 0x4000-0x7FFF | paged code
+---------------+
| 0x8000-0x???? | heap memory at bottom. stack memory at top
+---------------+
| 0x????-0xFFFF | cpm tpa code
+---------------+

I don't even have to define directly which functions are in which pages.

I'd prefer to get my forked z88dk commit merged upstream, but I don't need it to be as it's only 1 commit and I can easily rebase it. Using upstream z88dk you can also alter the stack offset using a special calling convention, but to do this I'd have to have a lot of extra z88dk specific function declarations, which are actually rather difficult to change for the flex and bison generated parser code I use.

Here's the javascript file I use to build for cp/m and msx machines:

https://github.com/pjshumphreys/querycs ... ild-cpm.js
nuc1e0n
Member
Posts: 49
Joined: Wed Jun 10, 2020 12:34 am

Re: Developing for MSX-DOS2 and >64kb

Post by nuc1e0n »

FYI I use these commands to test with msx dos2 :

Code: Select all

openmsx -machine National_FS-5000 -ext ram4mb -ext msxdos2 -diska ~/Projects/querycsv/env/cpm/MDOS22DE.DSK -diskb ~/Projects/querycsv/env/cpm/output/
and these ones for msxdos 1:

Code: Select all

cp MU.COM ./output/
openmsx -machine National_FS-5000 -ext ram4mb -diska ~/Projects/querycsv/env/cpm/MSXDOS.DSK -diskb ~/Projects/querycsv/env/cpm/output/
User avatar
dom
Well known member
Posts: 2072
Joined: Sun Jul 15, 2007 10:01 pm

Re: Developing for MSX-DOS2 and >64kb

Post by dom »

Thanks for the command lines - I've managed to get MU.COM hitting di/halt problem which is a little annoying so I can't test on MSXDOS1.

I can add that offset option in (I remember something similar was there for years but I removed it fairly recently!)

Auto-packing is quite a neat idea - I'd really like to do that "officially" because it would make life much easier rather than manually laying out. I'm wondering if it would be better to work at the object file level rather than the assembler level - I can see you've had to switch to -O0 to make it easier to work at assembler level? Though I can spy some complexities with static data either way.
nuc1e0n
Member
Posts: 49
Joined: Wed Jun 10, 2020 12:34 am

Re: Developing for MSX-DOS2 and >64kb

Post by nuc1e0n »

Personally, I would be very thankful if you could add the offset option in. I wouldn't need to keep maintaining my custom fork any longer :) But of course its not my code to make changes to :)

The __z88dk_params_offset(VV) calling convention is very useful for just a few functions that need a call trampoline, but for my code I have a large number of functions that I want to locate in banked memory which are generated from systems I cannot modify (flex, bison and gperf)
nuc1e0n
Member
Posts: 49
Joined: Wed Jun 10, 2020 12:34 am

Re: Developing for MSX-DOS2 and >64kb

Post by nuc1e0n »

As for complexities with auto-packing static data, yes there are quite a few. That's why I don't bank data in the stack or heap. My use case only needs a 'large text/code, small data/bss' solution. const data arrays can still be banked of course, which would be appropriate for level or sprite data in a game for example.

Even so I've still had to come up with special cases to cope with the problems, especially around calls to l_bsearch.
nuc1e0n
Member
Posts: 49
Joined: Wed Jun 10, 2020 12:34 am

Re: Developing for MSX-DOS2 and >64kb

Post by nuc1e0n »

I see now that a compilation parameter 'params-offset' has now been added (https://github.com/z88dk/z88dk/commit/5 ... b831899456). This is great, thank you very much :)

I realise this parameter is only really useful if using z88dk to compile C code to assembly source, and that anyone using it will need to write trampoline assembly code themselves to make use of the assembly output. However this is still really useful to me.

On another note, I've been trying to get my own CP/M and MSX project to compile once again (I broke a couple of things trying to get it to compile with Turbo C on the PC). In the process of doing this and now I'm using an up-to-date version of z88dk, I notice that file caching was added for CP/M last April.

I have another small change request I'd like to ask for regarding this caching. However the background information is a little lengthy so I'll start a separate thread for that.
megatronuk
Member
Posts: 34
Joined: Fri Jan 28, 2022 11:39 pm

Re: Developing for MSX-DOS2 and >64kb

Post by megatronuk »

Hi folks,

I've just updated my z88dk install to the latest Git (23rd October 2022) and the msx-dos banked example code (examples/banked) that was implemented earlier this year is no longer working.

The only change I have made is to alter the CFLAGS_msx_bank options - from the default ROM type:

CFLAGS_msx_bank = -subtype=rom -pragma-define:MAPPER_ASCII16

to

CFLAGS_msx_bank = -subtype=msxdos2 -pragma-define:CRT_DISABLELOADER=0

... as was discussed earlier in this thread for msxdos1/msxdos2 project types.

Instead of the previous behaviour of the 4 bank segments being loaded, and then the printed messages from each function, the startup code tries to load bank files 01 through to 1B and then exits with a segment allocation error.
banked_msx_.png
zcc reports the following version:
zcc - Frontend for the z88dk Cross-C Compiler - v20246-874e2e026-20221023
My local repo shows the following as the last commit:
commit 874e2e026d04d986487d2ccd488191d664d8fe12 (HEAD -> master, origin/master, origin/HEAD)
Merge: b8e67d6ab 7c2d9cf72
Author: Stefano <stefano_bodrato@hotmail.com>
Date: Fri Oct 21 16:28:50 2022 +0200

Merge pull request #2109 from zx70/master

1bit sound fixes (4bit for trs80, beeper for zxrom)
You do not have the required permissions to view the files attached to this post.
stefano
Well known member
Posts: 2137
Joined: Mon Jul 16, 2007 7:39 pm

Re: Developing for MSX-DOS2 and >64kb

Post by stefano »

this is very odd, I wonder how extending the library affects the bank switching
megatronuk
Member
Posts: 34
Joined: Fri Jan 28, 2022 11:39 pm

Re: Developing for MSX-DOS2 and >64kb

Post by megatronuk »

I've stripped my test case down to the bare minimum:

game.c:

Code: Select all

#include <stdio.h>
#include "banks.h"

int main(){	
	printf("Hello MSX world\n");
	func_bank1();
	return(0);
}
bank1.c:

Code: Select all

#include <stdio.h>
#include "banks.h"

#define BANK 1
#pragma bank 1

int func_bank1(){
	printf("Printing from bank[%d]\n",BANK);
	return(0);
}
banks.h:

Code: Select all

#ifndef BANKING_H
#define BANKING_H
extern int func_bank1() __banked;
#endif
Compilation is:

zcc -compiler sdcc +msx -m -subtype=msxdos2 -pragma-define:CRT_DISABLELOADER=0 --Werror -I./src -o src/game.o -c src/game.c
zcc -compiler sdcc +msx -m -subtype=msxdos2 -pragma-define:CRT_DISABLELOADER=0 --Werror -I./src -o src/bank1.o -c src/bank1.c
zcc +msx -m -o bin/game.com -subtype=msxdos2 -pragma-define:CRT_DISABLELOADER=0 -create-app src/game.o src/bank1.o

I copy game.com and copy/rename game_BANK_01.bin to BANK.01 on to my msxdos floppy image (I don't use the built game.img file, as I have a floppy prepped with other tools that I use on the emulated system).

I start up OpenMSX, either running MSX-DOS2 a rom, or just with MSX-DOS1 (with the addition of mu.com to supply the missing mapper routines - without it the game.com binary correctly shows the 'no segments available' error message), I get that output where it attempts to load BANK.01 through BANK.1B, runs out of segments and then exits. None of the code in main() runs.

Even if I comment out the func_bank1() call, remove any reference to banks.h and recompile with just game.c (i.e. revert it to a single instance of a main() with no dependencies on banked calls), the loader still wants to scan for and loader 01 through 1B. Only if we set CRT_DISABLELOADER=1, does the system seem stop scanning and loading loadable bank files that are not there.

Looks like possibly a regression somewhere along the lines has broken the msxdos banked loader.
User avatar
dom
Well known member
Posts: 2072
Joined: Sun Jul 15, 2007 10:01 pm

Re: Developing for MSX-DOS2 and >64kb

Post by dom »

I'll try to reproduce this another evening.

However, nothing has changed in the loader since the work back in February: https://github.com/z88dk/z88dk/blob/mas ... s.asm#L229

Given that it's printing the bank names, then it looks like open is returning a valid handle which seems to be unexpected given that open is read only.
pjshumphreys
Member
Posts: 66
Joined: Sat Feb 06, 2021 2:32 pm

Re: Developing for MSX-DOS2 and >64kb

Post by pjshumphreys »

This commit occurred last month, might it be related? https://github.com/z88dk/z88dk/commit/e ... 918f91128c

It might be worth checking whether the example works on the commit before.
User avatar
dom
Well known member
Posts: 2072
Joined: Sun Jul 15, 2007 10:01 pm

Re: Developing for MSX-DOS2 and >64kb

Post by dom »

That commit is for the ROM target, not MSX-DOS.

This was bugging me, so I've tried it out:

- Sony HB-F1XDJ
- Slot A = MSX-DOS2
- Slot B = 1MB mapper
Screenshot 2022-10-23 at 23.25.34.png
You do not have the required permissions to view the files attached to this post.
pjshumphreys
Member
Posts: 66
Joined: Sat Feb 06, 2021 2:32 pm

Re: Developing for MSX-DOS2 and >64kb

Post by pjshumphreys »

Code: Select all

That commit is for the ROM target, not MSX-DOS.
I would have thought so as well, but I've found it hard to tell what's related in other large projects in the past. If something is reported as not working as intended when it previously did, I find it's useful to start with what's close by and that has changed recently. Although this sample appears to work ok for yourself.
Last edited by pjshumphreys on Sun Oct 23, 2022 11:08 pm, edited 1 time in total.
megatronuk
Member
Posts: 34
Joined: Fri Jan 28, 2022 11:39 pm

Re: Developing for MSX-DOS2 and >64kb

Post by megatronuk »

Well in that case, I don't know what I've done, but I simply cannot get this to work any more. I was sure that it was working via MSX DOS 1 and MU.COM, but now I am not sure if my memory is playing up.

I simply can't find a working combination of MSX DOS 2 rom + bootable floppy to test it now. What I thought was a working image clearly isn't.
pjshumphreys
Member
Posts: 66
Joined: Sat Feb 06, 2021 2:32 pm

Re: Developing for MSX-DOS2 and >64kb

Post by pjshumphreys »

A bad merge to your local repo from upstream perhaps? I think msx.org has some msx-dos boot disks in case you need them.
pjshumphreys
Member
Posts: 66
Joined: Sat Feb 06, 2021 2:32 pm

Re: Developing for MSX-DOS2 and >64kb

Post by pjshumphreys »

There is this one on archive.org: https://archive.org/details/MSXDOS
User avatar
dom
Well known member
Posts: 2072
Joined: Sun Jul 15, 2007 10:01 pm

Re: Developing for MSX-DOS2 and >64kb

Post by dom »

megatronuk wrote: Sun Oct 23, 2022 11:08 pm Well in that case, I don't know what I've done, but I simply cannot get this to work any more. I was sure that it was working via MSX DOS 1 and MU.COM, but now I am not sure if my memory is playing up.
It looks like I couldn't (and still can't) get mu.com to run correctly - it hangs with a di; halt sequence.

The disc I use for MSX-DOS2 is this one:

Code: Select all

% md5 MSX-DOS\ v2.20\ \(1988\)\(Microsoft\)\[MegaRAM\].dsk
MD5 (MSX-DOS v2.20 (1988)(Microsoft)[MegaRAM].dsk) = 7496aca0f68fd7ece6831371667d2a31
No idea where I grabbed it from unfortunately.
megatronuk
Member
Posts: 34
Joined: Fri Jan 28, 2022 11:39 pm

Re: Developing for MSX-DOS2 and >64kb

Post by megatronuk »

It seems that in my recent testing I have been inadvertently using only an MSX-DOS 1 setup, or at least in every variation where only DOS 1 is being used (i.e. the ASCII DOS 2 rom extension and a MSX DOS 1 formatted floppy, with/without MU.COM), I get that BANK.01-1B behaviour.

I've found a DOS 2 formatted floppy, and with the ASCII DOS 2 rom extension I'm still not seeing the behaviour we had earlier this year, nor what you just posted now, Dom:
msxdos2_nomapper.png
Booting with the DOS 2 floppy, ASCII DOS 2 rom and 512kb mapper, I get the "cannot allocate segment" error. My brain is mush right now, I think I'll have to try this again later as I cannot work out what is wrong with that combination.

I was sure I had tried this on DOS 1 earlier this year.
You do not have the required permissions to view the files attached to this post.
pjshumphreys
Member
Posts: 66
Joined: Sat Feb 06, 2021 2:32 pm

Re: Developing for MSX-DOS2 and >64kb

Post by pjshumphreys »

Ah ha. My code uses mu.com but not the built in mapper functionality in z88dk (my code predates that).

If I don't have the msxdos2 extension plugged in mu.com also fails for me **unless** I copy mu.com to the boot disk. According to mu.com's english translated documentation:
2/ How to activate

In order to use MU, please copy MSXDOS.SYS, COMMAND.COM and MU.COM to an
MSX-DOS bootable disk (MSX-DOS formatted disk).
So it seems the driver needs to be in the proper place to work with only msxdos1.
pjshumphreys
Member
Posts: 66
Joined: Sat Feb 06, 2021 2:32 pm

Re: Developing for MSX-DOS2 and >64kb

Post by pjshumphreys »

Or maybe not. It says it works but attempting to call the mapper just hangs the machine. hmph. Now I'm keen to find out more of what's up with this driver. But not tonight.
megatronuk
Member
Posts: 34
Joined: Fri Jan 28, 2022 11:39 pm

Re: Developing for MSX-DOS2 and >64kb

Post by megatronuk »

Yes, the english text included with mu.com, and (I think) our understanding of what it does (provide MSX2 compatible mapper functionality on a MSX-DOS 1 environment) does seem at odds with how it is behaving. That, however, I can put aside as perhaps some slight difference in implementation (which I would hope we could resolve - the advantages of having z88dk banked code for MSX running in both DOS1 and DOS2 is huge).

What I cannot understand is why I am unable to replicate my earlier success with the examples/banked code from earlier in the year with a complete DOS2 setup. I suspect I'll have to pull out my real MSX2+ (which is a full DOS2 system with Nextor, mapper, etc) to remind myself.

[edit] I found a copy of the same bootable MSX-DOS 2.20 disk that Dom posted above, and with this version bank.com does indeed work as intended:
msxdos220_working.png
... yet on the other MSX DOS 2 disk, as shown in my previous post, it failed to allocate any segments. Same virtual machine spec, same extensions plugged in and exactly the same DOS component versions reported as my (failed) post previous to this one, but a different floppy image. In neither case using mu.com (which isn't/shouldn't-be necessary on a pure DOS2 system).
You do not have the required permissions to view the files attached to this post.
megatronuk
Member
Posts: 34
Joined: Fri Jan 28, 2022 11:39 pm

Re: Developing for MSX-DOS2 and >64kb

Post by megatronuk »

Reminder to self:

Look at autoexec.bat of any bootable disk you use.....

If it tries to load a ramdisk of more than the entire size of your mapper memory, then you're not going to have any free segments left for banked z88dk code. FFS.

I think that puts the MSX-DOS2 issue to bed. Just as I should have done instead of trying to debug things after 12 at night....

=|

The MSX-DOS1 situation definitely needs more looking at though. It's a huge advantage to be able to run on both DOS1 and 2; lots of people will have (or will prefer) to run DOS1 systems, but still have a cart that adds mapper memory to their system.
Post Reply