Return Array<T>

[Lelelo1]

I am noticing I get a message from setting the return type to Array<String> of a method, when compiling:

Array is not supported for C export, try using cpp.Pointer instead

When try to walk around using a List<T> I can compile, but the method's return type is given as HaxeObject

Is there some way I can construct a suitable array to be returned - and receiving array or list out from the haxe environment?

[haxiomic]

Updated answer: there isn't currently an easy way to expose Array safely, and Arrays of strings is a step harder. This is because C types are very basic, a string is just a pointer to some characters, whereas in haxe, Strings are objects with garbage collection details associated with them. Same goes for Arrays.

The solution is for me to implement a compatibility type that we convert to automatically when you return Array. Not sure when this will happen however

In general, because C is so simple it's best to only pass the most basic types you can

To answer your question for now: when passing to C we have to be careful that the haxe garbage collector isn't going to free the memory we've passed out in the background. The following will work but it's not ideal

static public function getHaxeArrayStr(length: Star<Int>) {
	var array = ['a', 'bbb', 'c'];
	Native.set(length, array.length);

	// we cannot pass this to C as-is, because it's not an array of C-friendly strings, it's an array of haxe string objects
	// instead we want to return a C array of C strings, aka const char **
	// we need some way to manage the memory, we could allocate here with malloc, but then who will free this memory in the future?

	// we cannot have Array<ConstCharStar> but we can have Array<Int64> and cast our pointer to Int64   
	var nativeArray = new Array<cpp.Int64>();
	for (i in 0...array.length) {
		var cStr = ConstCharStar.fromString(array[i]);
		var ptrInt64: cpp.Int64 = untyped __cpp__('reinterpret_cast<int64_t>({0})', cStr);
		nativeArray[i] = ptrInt64;
	}

	HaxeCBridge.retainHaxeObject(nativeArray);

	return cpp.Pointer.ofArray(nativeArray);
}

Then in C

int arrayLength = 0;
const char** array = HaxeLib_getHaxeArrayStr(&arrayLength);

[haxiomic]

• updated my earlier answer

[Lelelo1]

Interesting, thanks for the details.

Can I use structure?

[haxiomic]

Absolutely, here's how I plan on passing arrays in the future:

• Define type in C (this works for any array type, but we might want to generate type specialized versions like HaxeArrayInt

typedef struct HaxeArray {
	const void* ptr;
	int64 length;
} HaxeArray;

• When passing an array to C we:

  • Make any necessary type conversions, for haxe objects like Strings we need to convert to const char*
  • Get the pointer to the underlying data, using cpp.Pointer.ofArray(array)
  • Keep a ref to the array so the data doesn't get garbage collected, this is what HaxeCBridge.retainHaxeObject(array); does
  • Return a stack allocated struct of HaxeArray (example)

• When passing one of these HaxeArrays from C to Haxe, we:

  • Use cpp.NativeArray.setUnmanagedData() to create an Array from our ptr and length

[Lelelo1]

I assume this file is needed to define the return type:

haxe-c-bridge/test/unit/src/MessagePayload.h

Line 3 in 631cb58

?

.. or I get HaxeObject as return type in the generated header?

[haxiomic]

Exactly, the MessagePayload is an example of passing around a custom C struct in haxe, checkout how it's used in app.c and Main.hx

You'd be doing the same thing but with a new struct called HaxeArray

[Lelelo1]

Is there a possibility one can provide a type declaration in the HaxeArray struct:

typedef struct HaxeArray {
	const void* ptr;
	const Employee; // <---
	int64 length;
} HaxeArray;

... Employee struct:

typedef struct {
    char name[NAMESIZE];
    char sex;
} Person;

typedef struct {
    Person person;
    char job[JOBSIZE];
} Employee;

Can I then make a cast of ptr and initialize it to the given length when consuming the output?

HaxeArray fromHaxe; // from haxe

Employee employees[fromHaxe.length] = (some cast to array of Employee) fromHaxe.prt;

// use employees with type declaration

[haxiomic]

You can't store a type reference in C like that (it'll complain at the syntax level) but you could create an enum for this, like

enum ArrayType {
    EMPLOYEE,
    PERSON,
};

typedef struct HaxeArray {
	const void* ptr;
	const ArrayType arrayType;
	int length;
} HaxeArray;

Then we can know what array type we have if we switch on the arrayType field

[Lelelo1]

// above struct definition of employee

typedef struct HaxeArray {
	const void* ptr;
	Employee type; // <--- (corrected)
	int64 length;
} HaxeArray;

I read about both anonymous structs and nested typedef structs as well, in C.

From I own current use case - to pass an array of models out of haxe, I view this as a serialization area. It reminds be of web request, json, and react-native bridge. Meaning there is highlevel in Haxe, and low level C - and in the end you want higher programming features and data, after consuming C.

The type would be just a dummy variable not storing anything, but allowing casting of prt to it, when consuming C

[haxiomic]

The type field doesn't buy you anything I'm afraid – C does not store type information, that only exists at compile-time. So if you receive an array, where you don't know the type of the type field:

typedef struct HaxeArray {
	const void* ptr;
	? type;
	int64 length;
} HaxeArray;

You cannot tell type is an Employee unless you store something manually to indicate that

So you have two options

• know the array type when you pass it to C, ie: getEmployees(): HaxeArray_Employee
• store information that tells you about the type:

HaxeArray* array = getArray();
switch (array.typeEnum) {
    case EMPLOYEE:
    break;
...

[Lelelo1]

You cannot tell type is an Employee unless you store something manually to indicate that

Can I somewhere in haxe set the Employee type to the first element of the array, or assign a template value to it?

[haxiomic]

const void* ptr; points to the first element of the array, the trouble is, we have to store some additional information if you want to know what it points to. Types in C only exist at compile-time, so to pass type information around you have to do it manually by pairing values with types

[Lelelo1]

const void* ptr; points to the first element of the array

Ok, I thought that pointed to the whole value of the whole array!

I assume you can save the "standard types", like bool, float etc, (and methods?). But you can't declare struct (and nest them)?

[haxiomic]

Say you have a struct with an int field, if you lose the type information, ie you get a void pointer to this struct, there’s no way to be get that back - you just have a pointer to some random data, you don’t know where the fields start and end without the struct

you can cast this pointer to your original struct and it’ll work again, or you can cast to another struct and it’ll probably crash when you try to use it

You could try to examine the values that your pointer points to but there’s no real way to tell if these 4 bytes should be interpreted as an int or as a float or whatever else - so there’s no notion of saving type data for standard types either

[Lelelo1]

typedef struct MessagePayload {
	float someFloat; // type remains in generated header file
	char cStr[10]; // type remains in generated header file
} MessagePayload; // everything ok


typedef struct StructWithTypeInside { // variables that gets declared with the type becomes void pointers in the generated header file?
	MessagePayload messagePayload; // -----> ? messagePayload; // Or is the problem here - that type is lost?
} StructWithTypeInside 

[haxiomic]

The problem only is when you have some data passed to C where you’ve lost the type. So you don’t have to lose the type at all, you can have a function in haxe like getEmployees() and this could return a struct that contains a pointer to a set of Employees. So in your struct, const void* Ptr would become const Employee* ptr.

However, if you want to have something like getStuff(): Array, where you don’t explicitly say what the type is when you pass to C, then you’ve lose type information and you need to store it somewhere

[Lelelo1]

I think I might have understood, looking at MessagePayload again.

Then in haxe code it is exposed as a lambda function .

import cpp.Callable;
// ...
fnStruct: Callable<MessagePayload -> Void>

And in app.c, client

... the type stays

returning just MessagePayload would on the other hand cause the type to be lost.

import cpp.Callable;
// ...
fnStruct: MessagePayload

---

But you are saying also I can add a pointer to any array, of int, float or struct - inside MessagePayload..?

[haxiomic]

But you are saying also I can add a pointer to any array, of int, float or struct - inside MessagePayload..?

Yeah absolutely

returning just MessagePayload would on the other hand cause the type to be lost.

No, passing around the native C struct is no problem and the type isn't lost anywhere, see the externStruct() method in the tests:

In haxe:

static public function externStruct(v: MessagePayload, vStar: Star<MessagePayload>): MessagePayload {
	vStar.someFloat = 12.0;
	v.someFloat *= 2;
	return v;
}

Generated function in C

MessagePayload HaxeLib_externStruct(MessagePayload v, MessagePayload* vStar);

Here we pass around the C struct in different ways, modify it and pass it back, showing that it can happily cross to haxe and back with no issue.

---

I'll try to give an overview of the situation:

C types, including C structs are nice and simple and they can enter haxe-land and come back perfectly. However, C types are very limited, C does not have dynamic arrays with a length value for example. So you need to do something to pass a haxe array to C.

The closest thing to a dynamic array in C is a pointer, which stores the memory location of the first value of a series of a values stored next to each other in memory. So to access each value, you offset the pointer to find the memory location of the index you want.

This doesn't tell us how many items are in the array however, for that we need another variable. We can use a struct to group these to variables together, making it easier to pass to haxe and back.

Now something you may have been wondering is how do we know how much to offset the pointer by to seek to a specific index? Well for that we'd need to know the size of each element – so ints are 4 bytes, so to get to index 5 we'd offset by 5 * 4 bytes = 20 bytes. Fortunately C can do this for us by giving pointers explicit types. So we could have

int* integers = ...;
int v = integers[5];

Or with characters, which are 1 byte long

char* characters = ...;
char v = characters[5];

Or even

MessagePayload* messages = ...;
MessagePayload v = messages[5];

So a struct that could pass an array of integers from haxe to C could look like this

typedef struct ArrayInt {
	const int* ptr;
	const int64 length;
} ArrayInt;

where int can be replaced with any type supported in C.

Now, I understood that you were asking about passing an array where you don't know the type, i.e. Array. In which case, you can use const void* ptr; to create a pointer to an unknown type and instead store some value that indicates the type somewhere so you can cast this pointer to its correct type some point in the future. In general, there's not many good reasons to do this however so I wouldn't recommend it.

However, if you want to pass something like Array<Employee>, and Employee is a type we can represent in C (i.e a C struct, like MessagePayload) then you can define your employee array type

typedef struct ArrayEmployee {
	const Employee* ptr;
	const int64 length;
} ArrayEmployee;

And pass this struct around to haxe and back in the same way we pass around MessagePayload.

haxe-c-bridge could do this automatically for you, generating the struct types as required, but it's not something I've implemented yet

If you use this, make sure you understand about allocating on the stack vs allocating on the heap, and how stack memory is freed when it goes out of scope, where heap memory is never freed unless you explicitly free it yourself. Understanding those details is important to avoid memory leaks and crashes.

If all this makes sense so far, let me know how it goes and I can answer more questions and check things over for memory leak and whatnot

[Lelelo1]

It did clear out things for me.

• Type specific arrays
• Fixed size arrays

When it comes to keeping it in memory, can you go around the problem - by copying the array in the client/consuming application?

[haxiomic]

When it comes to keeping it in memory, can you go around the problem - by copying the array in the client/consuming application?

Absolutely, for example, you may stack allocate an array in haxe and return it, that way it'll be valid so long as the variable that references it in C stays in scope
https://en.wikipedia.org/wiki/Variable-length_array#:~:text=Implementation%5Bedit%5D-,C99,-%5Bedit%5D

Then you can copy it / do whatever you wish without worrying about allocation and haxe

*actually, I don't know if this works because haxe is running on a separate thread, hopefully the compile copies the array data automatically but you'll need to test to be sure!

[Lelelo1]

Do you heave any new ideas on how to handle this?
What I am looking for is to have shared logic (in Haxe) that can use used from swift, with typings. Is it simply a limitation in C language that makes this tricky?

[haxiomic]

Aye no way around this, it's simply C limitations making this tricky, so you need to use one of the solutions talked about here: passing the array length along with the array pointer

[singpolyma]

If I return cpp.Pointer.ofArray(someArray) from a function, it seems that haxe-c-bridge does not call retain on this automatically. I guess it just assumes if I am a Pointer that all bets are off? It seems like Pointer (vs RawPointer) is meant to be to something on the GC head and so should be auto retained still, but maybe I'm wrong?