ManagedObjectBackends for HSM #149

JBenda · 2022-05-09T09:36:13Z

Add JManagedBackends a Object Backeand which is at runtime composeble of multiple Object Backends.
JManagedBacends uses a policy module which can be used for HSM on that different object backends.

t-erxleben · 2023-01-16T15:08:12Z

include/core/jmanagedbackends.h

+	 **/
+	gboolean (*process_access)(gpointer policy_data, const gchar* namespace, const gchar* path, guint obj_id, guint tier, JObjectBackendAccessTypes access, gconstpointer data);
+
+	/// short processing period to match new object to a storage tier


Please fix documentation style.

t-erxleben · 2023-01-16T15:11:35Z

include/core/jmanagedbackends.h

+
+/**
+ * Fetches backend where a given object is stored. Also blocks migration for that object.
+ * Use j_backend_managed_object_open() to allow migration again. If the object not already exists


Did you mean j_backend_managed_close?

t-erxleben · 2023-01-16T15:14:19Z

include/core/jmanagedbackends.h

+gboolean j_backend_managed_object_migrate(JManagedBackends* this, const gchar* namespace, const gchar* path, guint dest);
+
+/**
+ * If resources a free migrates the specified object to the destination tier, if not return.


t-erxleben · 2023-01-16T15:15:33Z

include/core/jmanagedbackends.h

+
+/**
+ * Stops all migrations at all backends, used for maintenance.
+ * use j_backend_stack_unlock() to re-enable migration


j_backend_managed_unlock

t-erxleben · 2023-01-16T15:15:56Z

include/core/jmanagedbackends.h

+ * Stops all migrations at all backends, used for maintenance.
+ * use j_backend_stack_unlock() to re-enable migration
+ *
+ * \attention changing object locations or editing them might crashes the policy!


might crash

t-erxleben · 2023-01-16T15:18:30Z

lib/core/jmanagedbackends.c

+#include <sys/select.h>
+#include <sys/time.h>
+
+#define EXE(cmd, ...) \


I would like to have error checking macros on a global level instead of per module. It's probably better to move this one?

t-erxleben · 2023-01-16T15:19:33Z

lib/core/jmanagedbackends.c

+	gint write_access; ///< true if a write operation is scheduled or running
+} RWSpinLock;
+
+/// acquired a read lock


Please check doxygen comment style. (Also for following functions in this file.)

t-erxleben · 2023-01-16T15:31:20Z

lib/core/jmanagedbackends.c

+}
+
+/// proxy function
+gchar const* const* j_configuration_get_object_tiers(JConfiguration* config);


t-erxleben · 2023-01-16T15:34:49Z

lib/core/jmanagedbackends.c

+	    "failed to init kv for backend manager!");
+	this->kv_semantics = j_semantics_new(J_SEMANTICS_TEMPLATE_DEFAULT);
+
+	// this->log.filename = j_configuration_get_object_policy_log_file(config);


Why is this line commented?

t-erxleben · 2023-01-16T15:36:07Z

lib/core/jmanagedbackends.c

+	g_array_unref(this->rw_spin_locks);
+	g_module_close(this->module);
+	free(this);
+


There was a KV server module loaded in init. Should this one be freed as well?

t-erxleben · 2023-01-16T15:36:33Z

lib/core/jmanagedbackends.c

+	return TRUE;
+}
+
+/// \todo doku


Please add documentation (also below).

t-erxleben · 2023-01-16T15:41:06Z

include/core/jmanagedbackends.h

+/**
+ * Fetches backend where a given object is stored. Also blocks migration for that object.
+ * Use j_backend_managed_object_open() to allow migration again. If the object not already exists
+ * use j_backend_managed_object_create() to create it


open does also create if non-existent as implemented in *.c. Maybe this should be mentioned here. Otherwise someone could think open would fail on non-existant objects (and in worst-case use it as existence check).

t-erxleben · 2023-01-16T15:45:36Z

lib/core/jmanagedbackends.c

+	EXE(kv_get(this, namespace, path, &entry),
+	    "failed to fetch kventry");
+	return entry->backend_id;
+end:


Other parts of JULEA use _error as name for error handling label. Please change to prepare some unified error handling in JULEA.

Also in cases below.

t-erxleben · 2023-01-16T15:54:25Z

Looks good to me, overall.

Some remarks:

Why do you use a different logging mechanism as other parts of JULEA?
I really like the concept of error checking macros. Maybe we should move them to a global level?
Please complete the documentation and check again for typos/grammar in existing documentation.

t-erxleben · 2023-01-16T15:57:55Z

One general question:
How does this module relate to object distribution (for 'normal' objects) and JDistributedObject? As far as I have understood each JULEA server providing object storage will load this module to handle local tiering. Can it also handle different tiers on different JULEA servers?

JBenda · 2023-01-18T20:08:52Z

This module makes different ObjectBackends on one server appear to the outside as one ObjectServer, for distributed objects, they will be stripped between the servers as before, and are not effected by this module.

JBenda force-pushed the feature/ManagedObjectBackends branch from bcb85fd to 308b5bb Compare June 9, 2022 11:52

t-erxleben reviewed Jan 16, 2023

View reviewed changes

lib/core/jmanagedbackends.c Outdated

return TRUE;

}

/// \todo doku

Copy link

Contributor

t-erxleben Jan 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add documentation (also below).

t-erxleben reviewed Jan 16, 2023

View reviewed changes

JBenda added 12 commits January 19, 2023 11:51

Add ManagedBackendsfiles

6f32756

Documenting headerfile

0ba4530

Spellfixes and struct rename

02e2b0e

Documented and renamed headerfile

93e9a40

Rename functions in c

d0bcba5

Add object backend management configuration

c0cf740

Run autoformatter

083eb7c

Rebase to master

b567d6a

Add License for jmanagedeobjectbackend

b1f1546

Fix testes with new configuration

a914fb9

Refine commands and move Error Macros in jhelper.h

48c3d03

Run autoformater

ae005e6

JBenda force-pushed the feature/ManagedObjectBackends branch from 46934e2 to ae005e6 Compare January 19, 2023 10:55

JBenda added 6 commits January 19, 2023 16:01

fix warnings

24bd2d4

Optional HSM config

848e41e

Remove policy config from required config

f715d31

use glib logging in jmanagedbackend

db62a10

Merge branch 'master' into feature/ManagedObjectBackends

9f53797

Update PR to current master

decca51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ManagedObjectBackends for HSM #149

ManagedObjectBackends for HSM #149

JBenda commented May 9, 2022

t-erxleben Jan 16, 2023

t-erxleben Jan 16, 2023

t-erxleben Jan 16, 2023

t-erxleben Jan 16, 2023

t-erxleben Jan 16, 2023

t-erxleben Jan 16, 2023

t-erxleben Jan 16, 2023

t-erxleben Jan 16, 2023

t-erxleben Jan 16, 2023

t-erxleben Jan 16, 2023

t-erxleben Jan 16, 2023

t-erxleben Jan 16, 2023

t-erxleben Jan 16, 2023

t-erxleben Jan 16, 2023

t-erxleben commented Jan 16, 2023

t-erxleben commented Jan 16, 2023

JBenda commented Jan 18, 2023

ManagedObjectBackends for HSM #149

Are you sure you want to change the base?

ManagedObjectBackends for HSM #149

Conversation

JBenda commented May 9, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

t-erxleben commented Jan 16, 2023

t-erxleben commented Jan 16, 2023

JBenda commented Jan 18, 2023