Page 1 of 1

[z88dk-dev] zcc: llvm compilation and notemp gone

Posted: Sun Oct 23, 2016 8:47 am
by alvin
I'm still sitting on this commit until tomorrow when I'll have some more time to make sure compilation isn't broken.

* notemp gone

Intermediate files will always be written to the temp directory. Having the notemp option was complicating the m4 and llvm processing. When processing m4 files, zcc needs to copy the results of macro expansion to the original source directory immediately. This is because any produced .h, .inc, .asm, .c files have to be available from their original directories in order to properly resolve search paths and they have to be moved there right away so that succeeding source files in the current compile can make use of any produced .h and .inc files. Similarly with llvm compilation a new .c source file is produced that must also be immediately written to the original source directory to properly resolve search paths. So with the notemp option there, there could be a mixture of temporary files and newly generated permanent files mixed together in the original source directory which was really annoying. For debugging purposes the temp files can still be found in the temp direc!
when nocleanup is used.

Within zcc there are now three arrays that track filenames for each input source file, which serve essentially the same purposes as before: original_filenames[] contains the original filenames, filelist[] contains the name of the current working file and temporary_names[] contains the root name of temp files returned by tempnam(). Whenever m4 or llvm writes a file back to the original source directory, the original filename is changed to that new file. Also new is input source files are not copied to the temp directory immediately. Instead they are processed from their original locations so that search paths are properly resolved. Previously this was only done for .c files.

* llvm compilation

I've added llvm compilaton to zcc. I have no idea if it will work yet. You can read about it at Philip's page:

What zcc will do is make is transparent and automatic.

llvm requires headers that are free of any special attributes. So no callee, fastcall, preserve regs, whatever. This was easy to do for the new c library because headers for the new c lib are generated for each compiler from prototypes using a script. So this initial use of llvm will be for the new c lib for now. The set of llvm headers are in cvs already at z88dk/include./_DEVELOPMENT/standard and llvm will have have full access to the entire library.

For input .c files, clang is used to translate to llvm intermediate form .ll. Then llvm and the c-backend is used to optimize and emit a new .cbe.c file (another c file with longer extension to distinguish it from the original .c) This .c file is then compiled using sdcc per normal. At the sdcc step, sdcc is using the new c lib's sdcc headers which means all the fastcall, callee, preserve regs, etc are restored, meaning llvm compilation will not be comparatively impoverished.

The point of this is two-fold: (1) llvm+sdcc might be able to produce better code than sdcc alone. (2) we could start to use llvm to compile other languages to c which would then be compiled to z80 using sdcc.

There are a number of new options added:

-Cg"...." Add options to clang's command line
-Cv"...." Add options to llvm's command line
-clang Stop processing after clang has produced .ll files
-llvm Stop processing after llvm has produced .cbe.c files

zcc also tracks sdcc's --fsigned-char so that clang knows if chars are signed or not. zcc will also properly add include paths to clang via the normal -I option to zcc.

By default if you don't supply any arguments via -Cv to llvm, llvm will be invoked with "opt-3.8 -O2 -disable-simplify-libcalls -S " which turns on some optimization. clang will always have "-target sdcc-z80 -S -emit-llvm " added to its command line and llvm will always get "-disable-simplify-libcalls -S " which are required for things to function.

zcc is invoking the binaries "zclang" and "zllvm-cbe" so that we can avoid any conflicts by using our own private names.

I haven't actually tried it because I have to get these things compiled for windows so there are probably still outstanding problems but we can always cross fingers that it works the first time. If you're running linux it should be straightforward to compile zclang and zllvm-cbe.

Check out the vibrant tech community on one of the world's most
engaging tech sites,!

Posted: Sun Oct 23, 2016 9:52 pm
by alvin
I'm still sitting on this commit until tomorrow when I'll have some more time to make sure compilation isn't broken.
It's all been committed for tonight's build.

For llvm compilation, the clang and llvm-cbe binaries must be named "zclang" and "zllvm-cbe". Again I haven't actually tested llvm compilation yet because I don't have working binaries but everything has been done so that it should work AFAICT.

The new library has added new clib options "llvm_ix" and "llvm_iy" for its targets. These mirror sdcc_ix and sdcc_iy as underlying the llvm compile will be an sdcc compile. Just use the compiler as always, using sdcc command line flags if desired, with "-clib=llvm_ix" or "clib=llvm_iy"

Example compile using sdcc:
zcc +zx -vn -clib=sdcc_iy --max-allocs-per-node200000 test.c -o test -create-app

Example compile using llvm:
zcc +zx -vn -clib=llvm_iy --max-allocs-per-node200000 test.c -o test -create-app

I'd recommend running llvm compiles without "-vn" to see the compile steps taken, at least while this is in a debugging stage.
As mentioned in a previous message, you can communicate specific options to clang or llvm with the -Cg"..." and -Cv"...." command line options.

All the usual zcc behaviour is supported:
zcc +zx -vn -a -clib=llvm_iy --max-allocs-per-node200000 test.c

will stop after translation to asm. We have to be more specific now: it will stop after llvm and sdcc have translated the c to asm.

And there are the two additional flags mentioned in a past message that allow examination of the output generated by clang or llvm:
zcc +zx -vn -clang -clib=llvm_iy --max-allocs-per-node200000 test.c
zcc +zx -vn -llvm -clib=llvm_iy --max-allocs-per-node200000 test.c

These will stop after translation to .ll and .cbe.c respectively.

If you try the zclang and zllvm-cbe binaries built before I do, let me know if it's working!

Check out the vibrant tech community on one of the world's most
engaging tech sites,!

Posted: Sun Oct 23, 2016 9:57 pm
by alvin
One more thing: you will need to update zsdcc to the latest to grab some of the bug fixes associated with using llvm.

For windows I will be uploading a fresh zsdcc build in ... shortly.

Check out the vibrant tech community on one of the world's most
engaging tech sites,!

Posted: Mon Oct 24, 2016 6:57 pm
by alvin
I thought I'd better straighten out some of the naming early on. So starting in the Oct 25 build:

Compile lines that use clang are exactly like sdcc compile lines except either "-clib=clang_iy" or "-clib=clang_ix" are used to select clang instead of "-clib=sdcc_iy" or "-clib=sdcc_ix". All options applicable to sdcc should be added to the compile line as if it were an sdcc compile.

zcc +embedded -vn -SO3 -clib=sdcc_iy ?max-allocs-per-node200000 test.c -o test -lm -create-app
zcc +zx -vn -SO3 -clib=sdcc_iy --max-allocs-per-node200000 test.c -o test -create-app
zcc +cpm -vn -SO3 -clib=sdcc_iy --max-allocs-per-node200000 test.c -o test -create-app

zcc +embedded -vn -SO3 -clib=clang_iy ?max-allocs-per-node200000 test.c -o test -lm -create-app
zcc +zx -vn -SO3 -clib=clang_iy --max-allocs-per-node200000 test.c -o test -create-app
zcc +cpm -vn -SO3 -clib=clang_iy --max-allocs-per-node200000 test.c -o test -create-app

And all the usual options to zcc apply (brief overview ... ne_options ) with, as mentioned before, new additions:

-Cg"..." copy enclosed string to clang command line
-Cv"..." copy enclosed string to llvm command line
-clang stop after translation to .ll
-llvm stop after translation to .cbe.c

Check out the vibrant tech community on one of the world's most
engaging tech sites,!

Posted: Sun Oct 30, 2016 5:51 pm
by alvin
Getting llvm and the modified clang to compile on windows is turning into a bit of a nightmare.

llvm's own compile instructions for windows don't work - the result is a partly built kit and an llvm-config that produces incorrect path information. For that I used VS2015 but I've also been trying using MSYS2 and gcc but the VS2015 build gets furthest. I nearly got the clang-cfe to compile but it was missing either source code from llvm-3.8.0 or some part of the partly built llvm-3.8.0 kit that I had was missing. I've also been trying to use the llvm-3.8.0 pre-built binaries for windows but not getting further with that. I've got a couple more things to try but I think the ultimate answer might be having to cross compile from linux.

The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!

Posted: Sun Nov 06, 2016 5:27 pm
by alvin
I've been successfully compiling a few small programs using clang/llvm/sdcc using clang-3.9.1 and llvm-cbe-3.9.1, This is not using the c front end that Philip's made (this replaces clang-3.8.1) because for now I can't get it to compile. What we lose, among other things (I think), is proper type sizes in clang for the z80 target. I noticed that in the test compiles where llvm-cbe is generating ints that are 32-bit and so on. The following sdcc step can swallow some of this because those 32-bit ints often get passed into library functions that only accept 16-bit ints so sdcc is reducing their sizes.

The biggest test compile I tried was with eliza.c: ... xt%2Fplain

First an ordinary sdcc compile:
zcc +zx -vn -SO3 -clib=sdcc_iy --max-allocs-per-node200000 --opt-code-size eliza.c -o eliza -create-app
Output a tap file size 13329 bytes.

I'm using --opt-code-size for fairer comparison with clang because of the 32-bit ints that clang/llvm is generating in the output code. opt-code-size will add peephole rules that reduce code size for 32-bit longs in the clang compile.

Next with clang:
zcc +zx -vn -SO3 -clib=clang_iy --max-allocs-per-node200000 --opt-code-size eliza.c -o eliza2 -create-app

Errors appear at the sdcc compile step. zcc produces eliza.cbe.c which is the output from llvm-cbe before feeding into sdcc. There are a number of things that I had to fix in the .cbe.c to get the compile to complete:

* clang is processing the include files and llvm-cbe does not see them. What llvm-cbe does is produce prototypes for each required library function in the .cbe.c file. These prototypes are not correct first of all because they take 32-bit ints and so on, but secondly this is not what we want. We want the include files to remain intact in the .cbe.c file and eliminate those prototypes. The reason is zcc will use the sdcc headers in the sdcc compile step which adds back all the fastcall, callee and preserves_regs attributes. This makes a big difference in final code size in sdcc compiles. The manual fix was to delete the prototypes for library function in the .cbe.c file and manually add the #includes back into the .cbe.c file.

* llvm-cbe defined its own do-nothing structure for FILE* (and a number of other things) that caused sdcc to complain about incompatible pointer types. I had to manually add casts to a few points in the generated source code to allow sdcc to complete the compile.

* llvm-cbe added a call to "__builtin_unreachable();" in the source code. This appears after calls to "exit()" so llvm-cbe is probably using that to indicate to the compiler that the program does not return after calling exit(). I had to add a macro that turned that into nothing.

Lastly I had to add -lm to the compile line. llvm-cbe adds a bunch of static inlined functions that are helpers to deal with floating point numbers. Even though they aren't used, they are there. We'd have to get llvm-cbe to stop emitting those or we'd need to process the file to remove them and add them as library functions like we do now for sdcc and sccz80 helpers.

The next step to compile started from the modified .cbe.c file:

zcc +zx -vn -SO3 -clib=sdcc_iy --max-allocs-per-node200000 --opt-code-size eliza.cbe.c -o eliza2 -create-app
Output a tap file of size 15518 bytes. This was 1200 bytes larger than the normal sdcc file. Looking at the code there is still a lot of unnecessary 32-bit stuff in there, some of which was reduced by opt-code-size but there are still many opportunities for opt-code-size to be improved to reduce more of it. Hopefully Philip's cfe would get rid of these problems.

I then translated the .cbe.c file to assembler and tried assembling the result after manually removing the superfluous llvm float helpers:

zcc +zx -vn -a -SO3 -clib=sdcc_iy --max-allocs-per-node200000 --opt-code-size eliza.cbe.c
(remove llvm helpers from asm file)
zcc +zx -vn -clib=sdcc_iy eliza.cbe.c.asm -o eliza3 -create-app
The resulting tap file was 14872 bytes, now just 1500 bytes larger than the regular sdcc compile.

Anyway that's the update. There are still a few things to do to get clang running smoothly but I think if it can be made viable it will be a lot of fun to try to get other languages compiling via llvm-cbe.

Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.

Posted: Wed Nov 09, 2016 7:40 pm
by stefano
Not for beginners really, but intriguing :)

Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.