Possible problem with memory

ZX80, ZX 81, ZX Spectrum, TS2068 and other clones
Javi Perez
Member
Posts: 23
Joined: Wed Jul 29, 2015 9:07 am

Possible problem with memory

Post by Javi Perez »

Hi everyone,

I?m quite new to Z88DK so I want apologize upfront if this has already been discussed in this forum.

I?m currently working on a game for the ZXDEV 2015 compo http://zx-dev-2015.proboards.com/ and Z88Dk is the option I chose to build the game. I have started to experiment some weird and unexpected effects and my interpretation is that it?s due to memory issues.

Looking into the way memory space for dynamic allocation is allocated, I have the following:

Code: Select all

sbrk(40000, 10000);
If my program is stored from address location 32768, then I have less than 8KB for my program, and the binary is already larger than 8KB, so that means some memory overlapping is already happening.

Shall I change the store memory address, or shall I move the dynamic memory chunk? Is there any other way to sort this out?


Thanks in advance :)
Timmy
Well known member
Posts: 393
Joined: Sat Mar 10, 2012 4:18 pm

Post by Timmy »

The 32768 starting address is a default.

Spectrum games can start as low as 24000/25000. Add -zorg in your zcc command to start lower, example:

Code: Select all

zcc +zx somefile.c -zorg=25425
Of course, everything under 32768 will be contended.

You can also change the location of the SP at start by using

Code: Select all

#pragma output STACKPTR=53248
which is useful if you use sp1.

Note that using z88dk, you will always have problems with code size. So try to be efficient with your code.
alvin
Well known member
Posts: 1872
Joined: Mon Jul 16, 2007 7:39 pm

Post by alvin »

If weird stuff begins to happen, it's quite possible some things are overlapping in memory.

It's a good idea to keep a sketch tracking what your memory map looks like. When you have more experience with the tools, it's even better to plan what your memory map should look like :)

With the classic library and using sp1 you will have four memory regions to keep track of:

1. The program itself. By default it grows upward from 32768 but this can be changed with -zorg as Timmy pointed out. The reason why 32768 is the default start address is because that's the first byte of uncontended memory on the spectrum.

2. The stack. It will grow downward from its location. The STACKPTR pragma Timmy gave instructs the crt (that bit of code that runs before main) to move the stack pointer to address 53248. How much stack space is needed really depends on what your program is doing but 256 bytes is normally more than enough. So with that pragma in place, the stack can be given the address range 52992 through 53247.

3. Dynamic memory. sbrk(40000,10000) is allocating addresses 40000-49999 to the heap.

4. Memory reserved by sp1 for its data structures. The default has it using 53248 and up.


So your memory map is looking like this:

32768 - 39999 Program (about 7200 bytes available)
40000 - 49999 Dynamic memory (sbrk, 10k allotted)
52992 - 53247 Stack
53248 - 65535 Sp1

If the compiled binary you're getting is larger than about 7200 bytes then your program is overlapping the heap.

* With -zorg you can move the program downward in memory.

* There is a gap between the end of the dynamic memory block and the stack so there is room to move the dynamic memory block upward. Change the first parameter of sbrk to its new address. Another idea is to move this area far below 32768. You probably don't need 10k of heap too -- I expect you will be able to shrink this. If your own program doesn't use malloc then the only thing using the heap is sp1 to create the sprites. The amount of memory you actually need will depend on how many active sprites you are using and how big they are. A rough formula for the dynamic memory required by a sprite is: 26*R*C + 22 bytes were R and C are row height and column width in characters.

* sp1's default memory map looks like this: (I can't provide a link as sourceforge is still down)

; ADDRESS (HEX) LIBRARY DESCRIPTION
;
; f200 - ffff SP1.LIB horizontal rotation tables
; f000 - f1ff SP1.LIB tile array
; d200 - efff SP1.LIB update array for full size screen 32x24
; d1ff - d1ff SP1.LIB attribute buffer
; d1f7 - d1fe SP1.LIB pixel buffer
; d1ed - d1f6 SP1.LIB update list head - a dummy struct sp1_update acting as first in invalidated list
; * d1ef - d1f0 SP1.LIB update list tail pointer (inside dummy struct sp1_update)
; d1d4 - d1ec --free- 25 bytes free
; d1d1 - d1d3 ------- JP to im2 service routine (im2 table filled with 0xd1 bytes)
; d101 - d1d0 --free- 208 bytes
; d000 - d100 IM2.LIB im 2 vector table (257 bytes)

$d000 = 53248 which is the location of the stack according to the above arrangement.

If you're not using im2, you can get rid of the im2 table and jp. If you're not using all the horizontal rotation tables or tiles, you can use some of that memory as well but it will not be contiguous so only do that if you are desperate :) It can be useful to store variables up there.

sp1 can also be configured to use a smaller screen area than the full 32x24. If you do that the update array can be shrunk.
alvin
Well known member
Posts: 1872
Joined: Mon Jul 16, 2007 7:39 pm

Post by alvin »

Some broad suggestions to reduce code size:

http://www.z88dk.org/wiki/doku.php?id=t ... inary_size

#7 and up apply to the classic library as does #2.
Javi Perez
Member
Posts: 23
Joined: Wed Jul 29, 2015 9:07 am

Post by Javi Perez »

Many thanks Alvin and Timmy for your valuable help. I note that shrinking the dynamic area by 5KB with

Code: Select all

sbrk(45000, 5000);
fixes the memory issue and the program runs fine. However, starting in below 32768 in the contended area does not seem to fix the problem. Will keep checking...
alvin
Well known member
Posts: 1872
Joined: Mon Jul 16, 2007 7:39 pm

Post by alvin »

Javi Perez wrote:However, starting in below 32768 in the contended area does not seem to fix the problem. Will keep checking...
It should work. The only thing I can think of is you might be forgetting to change the RAND USR address? If you're compiling with -create-app, the tap created will have done that for you though.
Javi Perez
Member
Posts: 23
Joined: Wed Jul 29, 2015 9:07 am

Post by Javi Perez »

alvin wrote:It should work. The only thing I can think of is you might be forgetting to change the RAND USR address? If you're compiling with -create-app, the tap created will have done that for you though.
I use the below script to compile and build the tap:

Code: Select all

zcc +zx -vn %1.c -o %1 -lndos -lsp1 -lmalloc
appmake +zx -b %1 --org 32768
Using something like --org 25000 does not work :(
Timmy
Well known member
Posts: 393
Joined: Sat Mar 10, 2012 4:18 pm

Post by Timmy »

This is my script:

Code: Select all

zcc +zx -vn w1.c -o w1.bin -create-app -lsp1 -lmalloc -lim2 -zorg=25425
alvin
Well known member
Posts: 1872
Joined: Mon Jul 16, 2007 7:39 pm

Post by alvin »

The zorg and appmake org must match.

The zcc is making a binary that must be loaded and executed from the address indicated by zorg and appmake is making a basic loader that loads and RAND USRs the address indicated by --org.
Javi Perez
Member
Posts: 23
Joined: Wed Jul 29, 2015 9:07 am

Post by Javi Perez »

Timmy wrote:This is my script:

Code: Select all

zcc +zx -vn w1.c -o w1.bin -create-app -lsp1 -lmalloc -lim2 -zorg=25425
Thanks Timmy, but I believe I?m using an outdated version of zcc, this what I get if I use your script:

Code: Select all

Could not find parameter ZORG (not z88dk compiled?)
alvin
Well known member
Posts: 1872
Joined: Mon Jul 16, 2007 7:39 pm

Post by alvin »

That's an error from appmake indicating a symbol file couldn't be found. Was -lndos forgotten on the compile line?
Javi Perez
Member
Posts: 23
Joined: Wed Jul 29, 2015 9:07 am

Post by Javi Perez »

alvin wrote:That's an error from appmake indicating a symbol file couldn't be found. Was -lndos forgotten on the compile line?
-lndos is included, as follows:

Code: Select all

zcc +zx -vn map_test.c -o map_test.bin -create-app -lndos -lsp1 -lmalloc -lim2 -zorg=25425
alvin
Well known member
Posts: 1872
Joined: Mon Jul 16, 2007 7:39 pm

Post by alvin »

Javi Perez wrote:

Code: Select all

zcc +zx -vn map_test.c -o map_test.bin -create-app -lndos -lsp1 -lmalloc -lim2 -zorg=25425
What version of z88dk did you download?

The complaint is coming from using -create-app. You can still generate the tap file separately:

Code: Select all

zcc +zx -vn map_test.c -o map_test.bin -lndos -lsp1 -lmalloc -lim2 -zorg=25425
appmake +zx -b map_test.bin --org 25425
The two orgs must agree.


I believe a compile with the new c lib would be smaller but you would have to make a few changes to your source code. If you want to try that, post a link to your test code and I can fill in the changes.
Javi Perez
Member
Posts: 23
Joined: Wed Jul 29, 2015 9:07 am

Post by Javi Perez »

Comming back to this memory issue, the botom line then is that the maximum memory space I can use is 24,696 bytes provided that I shrink the dynamic memory to 5K bytes and I allocate the start of the program in the contended space right after the screen attributes area...
stefano
Well known member
Posts: 2151
Joined: Mon Jul 16, 2007 7:39 pm

Post by stefano »

Sometimes the -O3 options helps to save some space. You could also try to use "-startup=3".. it moves some global variable in the ZX printer buffer. Otherwise, if you feel adventurous, you might try the SDCC option in the _DEVELOPEMENT branch ;)

Let us know the compo results !
Javi Perez
Member
Posts: 23
Joined: Wed Jul 29, 2015 9:07 am

Post by Javi Perez »

stefano wrote:Sometimes the -O3 options helps to save some space. You could also try to use "-startup=3".. it moves some global variable in the ZX printer buffer. Otherwise, if you feel adventurous, you might try the SDCC option in the _DEVELOPEMENT branch ;)

Let us know the compo results !
Thanks Stefano, I ?ve managed to reduce 127 bytes following your suggestion!

Assuming there's no dynamic memory allocation, can I assume the that TAP file size in bytes is the same as the memory size in the Spectrum?
stefano
Well known member
Posts: 2151
Joined: Mon Jul 16, 2007 7:39 pm

Post by stefano »

It will be a little bigger because of the loader etc.. you should browse the tape file contents with an emulator or a tape tool to get the exact size.
You could compute the size difference once and keep it to subtract from the TAP file size in the next attemps.
alvin
Well known member
Posts: 1872
Joined: Mon Jul 16, 2007 7:39 pm

Post by alvin »

Javi Perez wrote:Comming back to this memory issue, the botom line then is that the maximum memory space I can use is 24,696 bytes provided that I shrink the dynamic memory to 5K bytes and I allocate the start of the program in the contended space right after the screen attributes area...
Yes that's about right. After that you can shrink the display area managed by sp1 (saving another ~2400 bytes is reasonable) and, with a little difficulty, you can decide only to move sprites to even horizontal pixel coordinates. The latter will allow using the memory reserved by the rotation lookup tables for pixel offsets 1,3,5,7 (512 bytes each, total 2048 bytes but not contiguous). I know I've mentioned this to you before; this is just for the record :) http://www.z88dk.org/wiki/doku.php?id=l ... es:sp1_ex1 discusses these details for the new clib.

You can achieve a lot in that memory space by making the game data-driven and careful planning but memory space will always be a challenge. The next step is to make those pressures go away by developing for the 128k spectrum. The simplest model is to have level-specific data and code in the extra banks and copy that into your main memory as needed. So the game in the main memory tracks global data and is only able to play the current 'level'. The new clib comes with lz77 data decompression so you can store compressed level data and code in the extra memory banks and decompress into main memory as needed.
Timmy
Well known member
Posts: 393
Joined: Sat Mar 10, 2012 4:18 pm

Post by Timmy »

There are some tricks to reduce compile size, of course.

For example, use more global variables instead of local.
Use your own simpler sprite routines instead of sp1.
Or drop some of your features.

Good luck.
Javi Perez
Member
Posts: 23
Joined: Wed Jul 29, 2015 9:07 am

Post by Javi Perez »

Thanks Alvin and Timmy for the useful hints. I'm making several changes like using more global variables, reducing the parameter list in functions, using unsigned data types, etc.

Unfortunately, building the program to start in the contended area, for example at 24000, does not help as the program won't simply start.

As a last option, I'll follow the suggestion to use the old splib routines instead of sp1.
stefano
Well known member
Posts: 2151
Joined: Mon Jul 16, 2007 7:39 pm

Post by stefano »

try to stay in a litte higher position, i.e. -zorg=24800
Javi Perez
Member
Posts: 23
Joined: Wed Jul 29, 2015 9:07 am

Post by Javi Perez »

Thanks Stefano for the suggestion, but allocation the program at any address below 32768 does not work, the program simply don?t execute!
alvin
Well known member
Posts: 1872
Joined: Mon Jul 16, 2007 7:39 pm

Post by alvin »

Javi Perez wrote:I'm making several changes like using more global variables, reducing the parameter list in functions, using unsigned data types, etc.
It's very easy to make the transition to global variables without making significant changes to the program:

Code: Select all

int function(int a, int b)
{
   int i, j, k;
   struct sp1_ss *s;

   ...
}
becomes:

Code: Select all

int function(int a, int b)
{
   static int i, j, k;
   static struct sp1_ss *s;

   ...
}
Instead of being allocated on the stack at runtime, the compiler will allocate those variables globally but give them complicated names so that they don't collide with other variables of the same C name. Even though static memory will be allocated for each such variable, this will always save memory because the code generated to access global variables is much cheaper than the code generated to access local variables. (The one exception might be the last local variable declared in a function; with sccz80 compiling it may be able to access it with pairs of pop/push functions which is two bytes rather than 3-byte static loads like "ld hl,(...)").

Unsigned types should always be used too. The Z80 code to deal with signed comparisons is clumsier than unsigned and having variables signed means the compiler must insert sign extension code when casting to larger bit widths. C contains integer promotion rules which means smaller types like char will be cast to integer before many operations and that will involve sign extension code for signed char.

If you have big functions with long parameter lists you can save code size by copying some parameters into static locals before using them:

Code: Select all

int function(int a, int b, int c, int d, int e, int f, int g)
{
...
}
becomes:

Code: Select all

int function(int a, int b, int c, int d, int e, int f, int g)
{
   static int la, lb, lc, ld, le, lf, lg;

   la = a; lb = b; lc = c; ld = d; le = e; lf = f; lg = g;

   // use la..lg instead of a..g here

   ...
}
If you're doing something like that you'll want to change frequently used parameters to static locals and probably leave alone one-off used parameters since in this case the extra assignment plus one access might be more expensive than a single local parameter lookup. A little experimentation will reveal what's best.

After this there is the idea of overlays but this requires some planning and clear thinking. You can create a global overlay struct and have storage space inside it that is shared by a lot of functions. But if you do that you have to make sure functions don't use the same variable space at the same time. This requires some planning from the outset so I wouldn't recommend retrofitting an existing program unless you do a rewrite.
Unfortunately, building the program to start in the contended area, for example at 24000, does not help as the program won't simply start.
Yes basic needs some space to actually load the program into memory. But what you can do is use basic to load the main program and then do a tape load from C to load data at 23296. http://www.z88dk.org/wiki/doku.php?id=l ... m#tape_i_o tape_load_block() is just a wrapper that jumps into the ROM to load a block of memory. You can do a headerless load and appmake is able to create a headerless tape block which you can just append to your tap file.

To do this you will have to place some of your global data at address 23296. One of the reasons z80asm was updated was to make this very easy to do. With the new clib you would just create an asm file like this:

lowmem.asm

Code: Select all

SECTION lowmem
org 23296

PUBLIC _a, _b

_a:  defs 100
_b:  defw 2
In your C program you can place variables there and use them as normal.

Code: Select all

extern unsigned char a[100];
extern int b;
When you compile, you include this file on the compile line:

zcc +zx -vn -clib=new test.c lowmem.asm -o test

The output will be two binaries:

test_CODE.bin
test_lowmem.bin

The first one is your program and the second one is the "lowmem" section which is your data at org 23296. You would create a tap file with test_lowmem.bin appended to the end and have your C program load it from tape first thing.


The classic library is not using sections but I think you can still do this by enabling sections on the command line! Give it a try:

zcc +zx -vn test.c lowmem.asm -o test -lndos -lsp1 -z80asm-sections

The output should be two binaries:

test.bin
test_lowmem.bin

The reason why this would work is there is no memory map so all stuff gets placed into the no-name section (hence "test.bin" rather than "test_CODE.bin" as in the new compile everything is being bundled into the CODE section). Except, of course, for the one defined section so it is output as a separate binary.


The other way to place stuff is to use the special @ syntax of sccz80 to put variables individually at specific memory locations. This can be tedious and error-prone however.
As a last option, I'll follow the suggestion to use the old splib routines instead of sp1.
I'm not so sure splib will always be smaller. Under splib sprite definitions are larger because the last column of sprites has to be an actual blank line whereas under sp1 the column can be empty. sp1 also separates the sprite draw code into individual parts whereas splib forces you to include all the sprite type draw code in each compile (you get draw code for OR, XOR, LOAD, MASK types whether you use them or not). The data structures are smaller but I'm not sure if it's a clear win in terms of memory use. It wouldn't hurt to try.

I'd also encourage you to try a new clib compile which I think will be at least a little bit smaller but also an sdcc compile which can sometimes be smaller as well.
alvin
Well known member
Posts: 1872
Joined: Mon Jul 16, 2007 7:39 pm

Post by alvin »

Javi Perez wrote:Thanks Stefano for the suggestion, but allocation the program at any address below 32768 does not work, the program simply don?t execute!
I'm not sure why you are having this problem.

Code: Select all

#include <stdio.h>

main()
{
   printf("Hello World\n");
}
If I compile this with:

zcc +zx -vn -zorg=28000 hello.c -o hello -lndos -create-app

The output is two files. The tap file loads and runs from 28000. The raw binary will also run if you load it into memory at address 28000 and manually enter "RAND USR 28000"

The only other thing I can think of is you are using your own tape loader, the start address is low, and you haven't CLEARed to make sure basic moves out of the way:

Code: Select all

10 CLEAR 27999: REM I'm not sure if 28000 has the right meaning
20 LOAD ""CODE 28000
30 RANDOMIZE USR 28000
Javi Perez
Member
Posts: 23
Joined: Wed Jul 29, 2015 9:07 am

Post by Javi Perez »

This is the output I get with that compile instruction:

Code: Select all

C:\z88dk\trabajos\13.bots_opt>zcc +zx -vn -zorg=28000 hello.c -o hello -lndos -create-app
        1 file(s) copied.
        1 file(s) copied.
Could not find parameter ZORG (not z88dk compiled?)
Building application code failed
And interestingly enough, I?m using the latest nightly build...

Now, if I use my compiling script:

Code: Select all

zcc +zx -vn hello.c -o hello -O3 -startup=3 -lndos -lsp1 -lmalloc
appmake +zx -b %1 --org 28000
And I run the Tap with Spectaculator, I can only see "Bytes: hello" and nothing else happens. So weird...
Post Reply