[RFC] psl1ght "v3"

[xerpi]

In order to modernize ps3toolchain and psl1ght, and thanks to the lessons learnt in https://github.com/vitasdk, I propose the following changes to psl1ght:

• The latest versions of newlib include all the necessary C headers (socket.h, etc) so we should remove them from psl1ght (they generate conflicts...), and the implementation of those functions should go to newlib.

• Add a NID database (like this one) and use it to generate the *_stub.a with a tool/script

• Make the function prototypes/enums/structs consistent: if the header is for an import named cell*, all the symbols in such header should begin with cell for functions, Cell for structs / enums, and CELL_ for constants (e.g., netCtlInit should be named cellNetCtlInitand the struct net_ctl_nat_infoshould be typedef struct CellNetCtlNatInfo { ... } CellNetCtlNatInfo;.

• All the syscalls should be in lowercase snake_case with the sys_ prefix.

As a PoC, I've already written a python script (I can port it to C if necessary) that generates an assembly file that gets compiled into the *_stub.a given a yml database. For example, given this input, it generates this output, which is easily compilable with ppu-as -mregnames -c cellCamera.s -o libcellCamera_stub.o.

This script can be useful to dump NIDs and libraries/modules information from an ELF file.

[zeldin]

Sounds nice.

I guess the renaming of typedefs and constants will break API compatibility; should we provide a backwards compat header to add the old names back as aliases? I'm guessing it's not that important and that we should not mess up the new code because of it.

[xerpi]

Well, my idea was that old homebrews can continue using the old version while new ones (or if they want to port) can use the newer version. Both versions could co-exist on the same system by changing the PSL1GHT environment variable when compiling.
Another important feature that we should target is the ability to generate PRXes.

EDIT: After thinking a bit more about PRXes:
A PRX needs to have:

• ELF's e_entry = 0 (or any value?, it isn't used)
• The first LOAD segment contains the sys_prx_module_info_t struct
• A PRXRELOC segment

So the options for the base ELF are:

1. Generate a PIE executable (DYN ELF).
2. Generate a regular executable ELF with the linker option -q(--emit-relocs). ← This is what VitaSDK does.

Then run a tool, let's call it psl1ght-prx-gen, that given the ELF and a list of exports, it walks the symtable, creates the sys_prx_module_info_tand converts the standard ELF relocations to the PRXRELOC segment.

For the imports, simply by linking with lib*_stub.a would add a .lib.stub section, then psl1ght-prx-gen will make the sys_prx_module_info_t::imports_start point to them.
Another utility of psl1ght-prx-gen (or psl1ght-libs-gen) would be to generate the .yml file of the exports, so other .sprx/.self can import those exports.

• psl1ght-prx-gen -> .yml -> psl1ght-libs-gen -> .a
• {.elf + exports.yml} -> psl1ght-prx-gen -> .sprx

[xerpi]

A lot of the explained ideas are inspired by: https://wiki.henkaku.xyz/vita/images/a/a2/Vita_SDK_specifications.pdf.

[grepwood]

3 cents anyone? Proper network stack support. I'm currently breaking my back porting github.com/ec-/Quake3e.

[miigotu]

@grepwood what do you need from us? Ill help if I can.

[grepwood]

@miigotu https://github.com/ec-/Quake3e/issues/55 we are missing a lot of headers normally found on Linux and BSD:

• net/if.h
• sys/ioctl.h
• ifaddrs.h
• sys/ipc.h
• sys/shm.h
• sys/mman.h (available in SPU part of the toolchain)
• dlfcn.h
• termios.h (includes wrong file from ps3dev, even if corrected it is massively underdeveloped)

[xerpi]

@grepwood I think those headers should be installed by newlib and not provided by this PSL1GHT. Check the first bullet of the first comment.

[grepwood]

@xerpi Right. I'll recompile ps3toolchain then. Is https://github.com/ps3dev/ps3toolchain this the correct repo?

[xerpi]

@grepwood I mean if we implement what I propose on this ticket, then those headers would be in newlib. Actually your comment at #67 (comment) started a totally off-topic discussion.

[miigotu]

@xerpi I'm with you all the way except the db/stub. How are those addresses generated?

[xerpi]

There's a list of NIDs here: https://www.psdevwiki.com/ps3/VSH_Exports

[bucanero]

I'm really happy to see that this RFC is being discussed again 👍

getting PSL1GHT to build .sprx binaries would be a dream, providing a full open-source solution for PS3 development. 😄

I'm not the most senior developer in these kind of topics and a lot of stuff might be way over my head, but I'll be happy to help. Even with just basic stuff like building the NID database, by going to the psdevwiki and copying all the stuff to the db.yml file.

[crystalct]

I'm new in PS3 homebrew development.. but i'm here.

[SuperDre]

Biggest problem I always have is setting up the whole enviroment, I'm from the kind of developer that just installs the IDE and starts developing. I once started with PS3 dev through a virtual machine, but after a couple of upgrades of Linux and every single time something was broken, I just gave up, also tried through the one on windows but that also went fubar after I had to reinstall windows, never got around of reinstalling it again.
I was working on a Tiny Tower port for the PS3 (didn't get really far yet as at that time an linux upgrade fubarred the whole setup).

[grepwood]

@xerpi I hate to rain on your parade but the headers I'm looking for to realize the Quake 3 port are still available only in the phoenix, rtems and Linux targets of newlib. Which one is psl1ght using?

[miigotu]

@xerpi I hate to rain on your parade but the headers I'm looking for to realize the Quake 3 port are still available only in the phoenix, rtems and Linux targets of newlib. Which one is psl1ght using?

Linux ppc64 - we would patch it to target the ppc64 arch if it wasn't supported, on a different branch. Then we would check out that branch for the submodule.

[dee12452]

Sorry to "bump" the thread, especially since I'm brand new to the scene, but I'd be more than happy to help make this happen. I was planning on working on porting/moving the net code over to newlib myself as a start (and to add some missing stubs in net code I need), but maybe having the NID DB and generating the .a's would be a better place to start.

[miigotu]

@dee12452 absolutely. Since we will have several things going on at the same time, just make the PRs to the develop branch.

[grepwood]

Before you start using newlib-3.3.0 you might want to see how every version of newlib since 2.5.0.20170519 is broken for the spu target https://bugs.gentoo.org/732206

[xerpi]

I've managed to get from my old laptop the folder containing the v3 experiments I was doing regarding stub generation. I have re-uploaded the files that had broken links (now those files should be permanent, I'm using gist.github):

For example, given this input, it generates this output, which is easily compilable with ppu-as -mregnames -c cellCamera.s -o libcellCamera_stub.o.

I have also uploaded a tool that can be helpful to examine the NIDs and libraries/modules of ELF files: nidreader.py.