Developing for MSX-DOS2 and >64kb
-
- Member
- Posts: 34
- Joined: Fri Jan 28, 2022 11:39 pm
Re: Developing for MSX-DOS2 and >64kb
It would certainly be more advantageous if both MSX-DOS2 and DOS1 users had the ability to use banked function/data calls (limited to machines with mapper support, obviously). I see that both 'MU.COM' and 'MSR.COM' appear to do the same thing - add DOS2/MSX2 compatible mapper calls on machines with the hardware support to do so, but without the need for DOS2.
It's not as elegant (nor automatic) as the bank pragma approach documented in the examples, but I can certainly see the advantages over the relatively heavy DOS2 implementation.
That could be a significant improvement, since we know that for all the advantages that DOS2 brings, it has big memory requirement over DOS1....
Out of interest, how do you build your 'prefix00.ovl' bank files?
It's not as elegant (nor automatic) as the bank pragma approach documented in the examples, but I can certainly see the advantages over the relatively heavy DOS2 implementation.
That could be a significant improvement, since we know that for all the advantages that DOS2 brings, it has big memory requirement over DOS1....
Out of interest, how do you build your 'prefix00.ovl' bank files?
Re: Developing for MSX-DOS2 and >64kb
Thanks for all the reports, I think I've now fixed some issues:
1. The disc format generated by -create-app now seems to work.
2. Only 4 banks were being loaded because I stopped the loop early for debug purposes
3. I've added log messages and abort the binary when the banks aren't successfully loaded
In terms of OpenMSX hardware, I was using the OpenMSX boosted MSXTurboR since it seemed to have the mapper in the right place. The fact that it doesn't work on so many machines means that we need to be smarter about selecting the mapper and fail back from the primary to another mapper?
Regarding MSX-DOS1 support, it looks like MU provides the same EXTBIOS entry points so the code in z88dk would be applicable there - it's also broadly the same as the code from nuc1e0n
1. The disc format generated by -create-app now seems to work.
2. Only 4 banks were being loaded because I stopped the loop early for debug purposes
3. I've added log messages and abort the binary when the banks aren't successfully loaded
In terms of OpenMSX hardware, I was using the OpenMSX boosted MSXTurboR since it seemed to have the mapper in the right place. The fact that it doesn't work on so many machines means that we need to be smarter about selecting the mapper and fail back from the primary to another mapper?
Regarding MSX-DOS1 support, it looks like MU provides the same EXTBIOS entry points so the code in z88dk would be applicable there - it's also broadly the same as the code from nuc1e0n
Re: Developing for MSX-DOS2 and >64kb
I forgot the screenshots:
Allocation failure to load on MSX2:
Running on an MSX1 without a mapper/exbios:
Successful:
Allocation failure to load on MSX2:
Running on an MSX1 without a mapper/exbios:
Successful:
You do not have the required permissions to view the files attached to this post.
Re: Developing for MSX-DOS2 and >64kb
I'm actually compiling them with my own fork of z88dk that has a new -cc-base command line option that I use to alter the stack offset of function local variables (here's a link to the only commit that's different to upstream z88dk: https://github.com/pjshumphreys/z88dk/c ... 1253a6126f)
Then I'm using a node.js script to orchestrate things (please don't judge. JavaScript is really good at manipulating big hash maps such as symbol tables). I use this script to slice up the assembly language output from z88dk into individual functions that I measure the binary sizes of and use a bin packing algorithm to pack the code into the minimum number of pages while still using information about which functions call which other ones to minimise page flipping.
The only things always kept in ram with my approach are the heap, stack, page flipping code and some of the C standard library code. All other code and constants (such as large arrays of constants and string literals) are paged in and out as necessary, using this memory map:
+---------------+
| 0x0000-0x00FF | cpm syscalls
+---------------+
| 0x0100-0x3FFF | paging code, cp/m or msx 2 file access routines, global variables and constants at bottom. extra heap memory at top
+---------------+
| 0x4000-0x7FFF | paged code
+---------------+
| 0x8000-0x???? | heap memory at bottom. stack memory at top
+---------------+
| 0x????-0xFFFF | cpm tpa code
+---------------+
I don't even have to define directly which functions are in which pages.
I'd prefer to get my forked z88dk commit merged upstream, but I don't need it to be as it's only 1 commit and I can easily rebase it. Using upstream z88dk you can also alter the stack offset using a special calling convention, but to do this I'd have to have a lot of extra z88dk specific function declarations, which are actually rather difficult to change for the flex and bison generated parser code I use.
Here's the javascript file I use to build for cp/m and msx machines:
https://github.com/pjshumphreys/querycs ... ild-cpm.js
Then I'm using a node.js script to orchestrate things (please don't judge. JavaScript is really good at manipulating big hash maps such as symbol tables). I use this script to slice up the assembly language output from z88dk into individual functions that I measure the binary sizes of and use a bin packing algorithm to pack the code into the minimum number of pages while still using information about which functions call which other ones to minimise page flipping.
The only things always kept in ram with my approach are the heap, stack, page flipping code and some of the C standard library code. All other code and constants (such as large arrays of constants and string literals) are paged in and out as necessary, using this memory map:
+---------------+
| 0x0000-0x00FF | cpm syscalls
+---------------+
| 0x0100-0x3FFF | paging code, cp/m or msx 2 file access routines, global variables and constants at bottom. extra heap memory at top
+---------------+
| 0x4000-0x7FFF | paged code
+---------------+
| 0x8000-0x???? | heap memory at bottom. stack memory at top
+---------------+
| 0x????-0xFFFF | cpm tpa code
+---------------+
I don't even have to define directly which functions are in which pages.
I'd prefer to get my forked z88dk commit merged upstream, but I don't need it to be as it's only 1 commit and I can easily rebase it. Using upstream z88dk you can also alter the stack offset using a special calling convention, but to do this I'd have to have a lot of extra z88dk specific function declarations, which are actually rather difficult to change for the flex and bison generated parser code I use.
Here's the javascript file I use to build for cp/m and msx machines:
https://github.com/pjshumphreys/querycs ... ild-cpm.js
Re: Developing for MSX-DOS2 and >64kb
FYI I use these commands to test with msx dos2 :
and these ones for msxdos 1:
Code: Select all
openmsx -machine National_FS-5000 -ext ram4mb -ext msxdos2 -diska ~/Projects/querycsv/env/cpm/MDOS22DE.DSK -diskb ~/Projects/querycsv/env/cpm/output/
Code: Select all
cp MU.COM ./output/
openmsx -machine National_FS-5000 -ext ram4mb -diska ~/Projects/querycsv/env/cpm/MSXDOS.DSK -diskb ~/Projects/querycsv/env/cpm/output/
Re: Developing for MSX-DOS2 and >64kb
Thanks for the command lines - I've managed to get MU.COM hitting di/halt problem which is a little annoying so I can't test on MSXDOS1.
I can add that offset option in (I remember something similar was there for years but I removed it fairly recently!)
Auto-packing is quite a neat idea - I'd really like to do that "officially" because it would make life much easier rather than manually laying out. I'm wondering if it would be better to work at the object file level rather than the assembler level - I can see you've had to switch to -O0 to make it easier to work at assembler level? Though I can spy some complexities with static data either way.
I can add that offset option in (I remember something similar was there for years but I removed it fairly recently!)
Auto-packing is quite a neat idea - I'd really like to do that "officially" because it would make life much easier rather than manually laying out. I'm wondering if it would be better to work at the object file level rather than the assembler level - I can see you've had to switch to -O0 to make it easier to work at assembler level? Though I can spy some complexities with static data either way.
Re: Developing for MSX-DOS2 and >64kb
Personally, I would be very thankful if you could add the offset option in. I wouldn't need to keep maintaining my custom fork any longer But of course its not my code to make changes to
The __z88dk_params_offset(VV) calling convention is very useful for just a few functions that need a call trampoline, but for my code I have a large number of functions that I want to locate in banked memory which are generated from systems I cannot modify (flex, bison and gperf)
The __z88dk_params_offset(VV) calling convention is very useful for just a few functions that need a call trampoline, but for my code I have a large number of functions that I want to locate in banked memory which are generated from systems I cannot modify (flex, bison and gperf)
Re: Developing for MSX-DOS2 and >64kb
As for complexities with auto-packing static data, yes there are quite a few. That's why I don't bank data in the stack or heap. My use case only needs a 'large text/code, small data/bss' solution. const data arrays can still be banked of course, which would be appropriate for level or sprite data in a game for example.
Even so I've still had to come up with special cases to cope with the problems, especially around calls to l_bsearch.
Even so I've still had to come up with special cases to cope with the problems, especially around calls to l_bsearch.
Re: Developing for MSX-DOS2 and >64kb
I see now that a compilation parameter 'params-offset' has now been added (https://github.com/z88dk/z88dk/commit/5 ... b831899456). This is great, thank you very much
I realise this parameter is only really useful if using z88dk to compile C code to assembly source, and that anyone using it will need to write trampoline assembly code themselves to make use of the assembly output. However this is still really useful to me.
On another note, I've been trying to get my own CP/M and MSX project to compile once again (I broke a couple of things trying to get it to compile with Turbo C on the PC). In the process of doing this and now I'm using an up-to-date version of z88dk, I notice that file caching was added for CP/M last April.
I have another small change request I'd like to ask for regarding this caching. However the background information is a little lengthy so I'll start a separate thread for that.
I realise this parameter is only really useful if using z88dk to compile C code to assembly source, and that anyone using it will need to write trampoline assembly code themselves to make use of the assembly output. However this is still really useful to me.
On another note, I've been trying to get my own CP/M and MSX project to compile once again (I broke a couple of things trying to get it to compile with Turbo C on the PC). In the process of doing this and now I'm using an up-to-date version of z88dk, I notice that file caching was added for CP/M last April.
I have another small change request I'd like to ask for regarding this caching. However the background information is a little lengthy so I'll start a separate thread for that.
-
- Member
- Posts: 34
- Joined: Fri Jan 28, 2022 11:39 pm
Re: Developing for MSX-DOS2 and >64kb
Hi folks,
I've just updated my z88dk install to the latest Git (23rd October 2022) and the msx-dos banked example code (examples/banked) that was implemented earlier this year is no longer working.
The only change I have made is to alter the CFLAGS_msx_bank options - from the default ROM type:
CFLAGS_msx_bank = -subtype=rom -pragma-define:MAPPER_ASCII16
to
CFLAGS_msx_bank = -subtype=msxdos2 -pragma-define:CRT_DISABLELOADER=0
... as was discussed earlier in this thread for msxdos1/msxdos2 project types.
Instead of the previous behaviour of the 4 bank segments being loaded, and then the printed messages from each function, the startup code tries to load bank files 01 through to 1B and then exits with a segment allocation error.
zcc reports the following version:
I've just updated my z88dk install to the latest Git (23rd October 2022) and the msx-dos banked example code (examples/banked) that was implemented earlier this year is no longer working.
The only change I have made is to alter the CFLAGS_msx_bank options - from the default ROM type:
CFLAGS_msx_bank = -subtype=rom -pragma-define:MAPPER_ASCII16
to
CFLAGS_msx_bank = -subtype=msxdos2 -pragma-define:CRT_DISABLELOADER=0
... as was discussed earlier in this thread for msxdos1/msxdos2 project types.
Instead of the previous behaviour of the 4 bank segments being loaded, and then the printed messages from each function, the startup code tries to load bank files 01 through to 1B and then exits with a segment allocation error.
zcc reports the following version:
My local repo shows the following as the last commit:zcc - Frontend for the z88dk Cross-C Compiler - v20246-874e2e026-20221023
commit 874e2e026d04d986487d2ccd488191d664d8fe12 (HEAD -> master, origin/master, origin/HEAD)
Merge: b8e67d6ab 7c2d9cf72
Author: Stefano <stefano_bodrato@hotmail.com>
Date: Fri Oct 21 16:28:50 2022 +0200
Merge pull request #2109 from zx70/master
1bit sound fixes (4bit for trs80, beeper for zxrom)
You do not have the required permissions to view the files attached to this post.
Re: Developing for MSX-DOS2 and >64kb
this is very odd, I wonder how extending the library affects the bank switching
-
- Member
- Posts: 34
- Joined: Fri Jan 28, 2022 11:39 pm
Re: Developing for MSX-DOS2 and >64kb
I've stripped my test case down to the bare minimum:
game.c:
bank1.c:
banks.h:
Compilation is:
zcc -compiler sdcc +msx -m -subtype=msxdos2 -pragma-define:CRT_DISABLELOADER=0 --Werror -I./src -o src/game.o -c src/game.c
zcc -compiler sdcc +msx -m -subtype=msxdos2 -pragma-define:CRT_DISABLELOADER=0 --Werror -I./src -o src/bank1.o -c src/bank1.c
zcc +msx -m -o bin/game.com -subtype=msxdos2 -pragma-define:CRT_DISABLELOADER=0 -create-app src/game.o src/bank1.o
I copy game.com and copy/rename game_BANK_01.bin to BANK.01 on to my msxdos floppy image (I don't use the built game.img file, as I have a floppy prepped with other tools that I use on the emulated system).
I start up OpenMSX, either running MSX-DOS2 a rom, or just with MSX-DOS1 (with the addition of mu.com to supply the missing mapper routines - without it the game.com binary correctly shows the 'no segments available' error message), I get that output where it attempts to load BANK.01 through BANK.1B, runs out of segments and then exits. None of the code in main() runs.
Even if I comment out the func_bank1() call, remove any reference to banks.h and recompile with just game.c (i.e. revert it to a single instance of a main() with no dependencies on banked calls), the loader still wants to scan for and loader 01 through 1B. Only if we set CRT_DISABLELOADER=1, does the system seem stop scanning and loading loadable bank files that are not there.
Looks like possibly a regression somewhere along the lines has broken the msxdos banked loader.
game.c:
Code: Select all
#include <stdio.h>
#include "banks.h"
int main(){
printf("Hello MSX world\n");
func_bank1();
return(0);
}
Code: Select all
#include <stdio.h>
#include "banks.h"
#define BANK 1
#pragma bank 1
int func_bank1(){
printf("Printing from bank[%d]\n",BANK);
return(0);
}
Code: Select all
#ifndef BANKING_H
#define BANKING_H
extern int func_bank1() __banked;
#endif
zcc -compiler sdcc +msx -m -subtype=msxdos2 -pragma-define:CRT_DISABLELOADER=0 --Werror -I./src -o src/game.o -c src/game.c
zcc -compiler sdcc +msx -m -subtype=msxdos2 -pragma-define:CRT_DISABLELOADER=0 --Werror -I./src -o src/bank1.o -c src/bank1.c
zcc +msx -m -o bin/game.com -subtype=msxdos2 -pragma-define:CRT_DISABLELOADER=0 -create-app src/game.o src/bank1.o
I copy game.com and copy/rename game_BANK_01.bin to BANK.01 on to my msxdos floppy image (I don't use the built game.img file, as I have a floppy prepped with other tools that I use on the emulated system).
I start up OpenMSX, either running MSX-DOS2 a rom, or just with MSX-DOS1 (with the addition of mu.com to supply the missing mapper routines - without it the game.com binary correctly shows the 'no segments available' error message), I get that output where it attempts to load BANK.01 through BANK.1B, runs out of segments and then exits. None of the code in main() runs.
Even if I comment out the func_bank1() call, remove any reference to banks.h and recompile with just game.c (i.e. revert it to a single instance of a main() with no dependencies on banked calls), the loader still wants to scan for and loader 01 through 1B. Only if we set CRT_DISABLELOADER=1, does the system seem stop scanning and loading loadable bank files that are not there.
Looks like possibly a regression somewhere along the lines has broken the msxdos banked loader.
Re: Developing for MSX-DOS2 and >64kb
I'll try to reproduce this another evening.
However, nothing has changed in the loader since the work back in February: https://github.com/z88dk/z88dk/blob/mas ... s.asm#L229
Given that it's printing the bank names, then it looks like open is returning a valid handle which seems to be unexpected given that open is read only.
However, nothing has changed in the loader since the work back in February: https://github.com/z88dk/z88dk/blob/mas ... s.asm#L229
Given that it's printing the bank names, then it looks like open is returning a valid handle which seems to be unexpected given that open is read only.
-
- Member
- Posts: 80
- Joined: Sat Feb 06, 2021 2:32 pm
Re: Developing for MSX-DOS2 and >64kb
This commit occurred last month, might it be related? https://github.com/z88dk/z88dk/commit/e ... 918f91128c
It might be worth checking whether the example works on the commit before.
It might be worth checking whether the example works on the commit before.
Re: Developing for MSX-DOS2 and >64kb
That commit is for the ROM target, not MSX-DOS.
This was bugging me, so I've tried it out:
- Sony HB-F1XDJ
- Slot A = MSX-DOS2
- Slot B = 1MB mapper
This was bugging me, so I've tried it out:
- Sony HB-F1XDJ
- Slot A = MSX-DOS2
- Slot B = 1MB mapper
You do not have the required permissions to view the files attached to this post.
-
- Member
- Posts: 80
- Joined: Sat Feb 06, 2021 2:32 pm
Re: Developing for MSX-DOS2 and >64kb
Code: Select all
That commit is for the ROM target, not MSX-DOS.
Last edited by pjshumphreys on Sun Oct 23, 2022 11:08 pm, edited 1 time in total.
-
- Member
- Posts: 34
- Joined: Fri Jan 28, 2022 11:39 pm
Re: Developing for MSX-DOS2 and >64kb
Well in that case, I don't know what I've done, but I simply cannot get this to work any more. I was sure that it was working via MSX DOS 1 and MU.COM, but now I am not sure if my memory is playing up.
I simply can't find a working combination of MSX DOS 2 rom + bootable floppy to test it now. What I thought was a working image clearly isn't.
I simply can't find a working combination of MSX DOS 2 rom + bootable floppy to test it now. What I thought was a working image clearly isn't.
-
- Member
- Posts: 80
- Joined: Sat Feb 06, 2021 2:32 pm
Re: Developing for MSX-DOS2 and >64kb
A bad merge to your local repo from upstream perhaps? I think msx.org has some msx-dos boot disks in case you need them.
-
- Member
- Posts: 80
- Joined: Sat Feb 06, 2021 2:32 pm
Re: Developing for MSX-DOS2 and >64kb
There is this one on archive.org: https://archive.org/details/MSXDOS
Re: Developing for MSX-DOS2 and >64kb
It looks like I couldn't (and still can't) get mu.com to run correctly - it hangs with a di; halt sequence.megatronuk wrote: ↑Sun Oct 23, 2022 11:08 pm Well in that case, I don't know what I've done, but I simply cannot get this to work any more. I was sure that it was working via MSX DOS 1 and MU.COM, but now I am not sure if my memory is playing up.
The disc I use for MSX-DOS2 is this one:
Code: Select all
% md5 MSX-DOS\ v2.20\ \(1988\)\(Microsoft\)\[MegaRAM\].dsk
MD5 (MSX-DOS v2.20 (1988)(Microsoft)[MegaRAM].dsk) = 7496aca0f68fd7ece6831371667d2a31
-
- Member
- Posts: 34
- Joined: Fri Jan 28, 2022 11:39 pm
Re: Developing for MSX-DOS2 and >64kb
It seems that in my recent testing I have been inadvertently using only an MSX-DOS 1 setup, or at least in every variation where only DOS 1 is being used (i.e. the ASCII DOS 2 rom extension and a MSX DOS 1 formatted floppy, with/without MU.COM), I get that BANK.01-1B behaviour.
I've found a DOS 2 formatted floppy, and with the ASCII DOS 2 rom extension I'm still not seeing the behaviour we had earlier this year, nor what you just posted now, Dom:
Booting with the DOS 2 floppy, ASCII DOS 2 rom and 512kb mapper, I get the "cannot allocate segment" error. My brain is mush right now, I think I'll have to try this again later as I cannot work out what is wrong with that combination.
I was sure I had tried this on DOS 1 earlier this year.
I've found a DOS 2 formatted floppy, and with the ASCII DOS 2 rom extension I'm still not seeing the behaviour we had earlier this year, nor what you just posted now, Dom:
Booting with the DOS 2 floppy, ASCII DOS 2 rom and 512kb mapper, I get the "cannot allocate segment" error. My brain is mush right now, I think I'll have to try this again later as I cannot work out what is wrong with that combination.
I was sure I had tried this on DOS 1 earlier this year.
You do not have the required permissions to view the files attached to this post.
-
- Member
- Posts: 80
- Joined: Sat Feb 06, 2021 2:32 pm
Re: Developing for MSX-DOS2 and >64kb
Ah ha. My code uses mu.com but not the built in mapper functionality in z88dk (my code predates that).
If I don't have the msxdos2 extension plugged in mu.com also fails for me **unless** I copy mu.com to the boot disk. According to mu.com's english translated documentation:
If I don't have the msxdos2 extension plugged in mu.com also fails for me **unless** I copy mu.com to the boot disk. According to mu.com's english translated documentation:
So it seems the driver needs to be in the proper place to work with only msxdos1.2/ How to activate
In order to use MU, please copy MSXDOS.SYS, COMMAND.COM and MU.COM to an
MSX-DOS bootable disk (MSX-DOS formatted disk).
-
- Member
- Posts: 80
- Joined: Sat Feb 06, 2021 2:32 pm
Re: Developing for MSX-DOS2 and >64kb
Or maybe not. It says it works but attempting to call the mapper just hangs the machine. hmph. Now I'm keen to find out more of what's up with this driver. But not tonight.
-
- Member
- Posts: 34
- Joined: Fri Jan 28, 2022 11:39 pm
Re: Developing for MSX-DOS2 and >64kb
Yes, the english text included with mu.com, and (I think) our understanding of what it does (provide MSX2 compatible mapper functionality on a MSX-DOS 1 environment) does seem at odds with how it is behaving. That, however, I can put aside as perhaps some slight difference in implementation (which I would hope we could resolve - the advantages of having z88dk banked code for MSX running in both DOS1 and DOS2 is huge).
What I cannot understand is why I am unable to replicate my earlier success with the examples/banked code from earlier in the year with a complete DOS2 setup. I suspect I'll have to pull out my real MSX2+ (which is a full DOS2 system with Nextor, mapper, etc) to remind myself.
[edit] I found a copy of the same bootable MSX-DOS 2.20 disk that Dom posted above, and with this version bank.com does indeed work as intended:
... yet on the other MSX DOS 2 disk, as shown in my previous post, it failed to allocate any segments. Same virtual machine spec, same extensions plugged in and exactly the same DOS component versions reported as my (failed) post previous to this one, but a different floppy image. In neither case using mu.com (which isn't/shouldn't-be necessary on a pure DOS2 system).
What I cannot understand is why I am unable to replicate my earlier success with the examples/banked code from earlier in the year with a complete DOS2 setup. I suspect I'll have to pull out my real MSX2+ (which is a full DOS2 system with Nextor, mapper, etc) to remind myself.
[edit] I found a copy of the same bootable MSX-DOS 2.20 disk that Dom posted above, and with this version bank.com does indeed work as intended:
... yet on the other MSX DOS 2 disk, as shown in my previous post, it failed to allocate any segments. Same virtual machine spec, same extensions plugged in and exactly the same DOS component versions reported as my (failed) post previous to this one, but a different floppy image. In neither case using mu.com (which isn't/shouldn't-be necessary on a pure DOS2 system).
You do not have the required permissions to view the files attached to this post.
-
- Member
- Posts: 34
- Joined: Fri Jan 28, 2022 11:39 pm
Re: Developing for MSX-DOS2 and >64kb
Reminder to self:
Look at autoexec.bat of any bootable disk you use.....
If it tries to load a ramdisk of more than the entire size of your mapper memory, then you're not going to have any free segments left for banked z88dk code. FFS.
I think that puts the MSX-DOS2 issue to bed. Just as I should have done instead of trying to debug things after 12 at night....
The MSX-DOS1 situation definitely needs more looking at though. It's a huge advantage to be able to run on both DOS1 and 2; lots of people will have (or will prefer) to run DOS1 systems, but still have a cart that adds mapper memory to their system.
Look at autoexec.bat of any bootable disk you use.....
If it tries to load a ramdisk of more than the entire size of your mapper memory, then you're not going to have any free segments left for banked z88dk code. FFS.
I think that puts the MSX-DOS2 issue to bed. Just as I should have done instead of trying to debug things after 12 at night....
The MSX-DOS1 situation definitely needs more looking at though. It's a huge advantage to be able to run on both DOS1 and 2; lots of people will have (or will prefer) to run DOS1 systems, but still have a cart that adds mapper memory to their system.