bitsquid_dt/p2p

Fork 0

generated from bitsquid_dt/dt-plugin-template

API design #2

New issue

Open

opened 2025-05-04 16:06:50 +02:00 by lucas · 3 comments

lucas commented

2025-05-04 16:06:50 +02:00

Owner

The overall API should be modeled after the engine's network features:

objects that transparently synchronize state to registered subscribers
RPCs that trigger behavior on the subscribed targets

It should also be modeled to fit the more specific requirements of mods as consumers:

lobbies/rooms to segment peers into the game session they are in
- should be implemented for both missions and the hub
- though for the hub, the question is whether it still only applies to the peers that are also in the same game session,
  or if it should combine all hubs
opt-in by default, such that peers negotiate their installed mods and automatically (de)activate based on mod presence
self-describing message encoding that can be provided at runtime
- ideally we would have tooling that would allow for a compilation phase during mod installation
- in liue of that, it needs to be a format passed from the mod at runtime
- preferably some string type like SJSON that can be parsed easily at runtime, but could also be a custom format based on Lua tables

Implementation details:

https://libp2p.io/

The overall API should be modeled after the engine's network features: - objects that transparently synchronize state to registered subscribers - RPCs that trigger behavior on the subscribed targets It should also be modeled to fit the more specific requirements of mods as consumers: - lobbies/rooms to segment peers into the game session they are in - should be implemented for both missions and the hub - though for the hub, the question is whether it still only applies to the peers that are also in the same game session, or if it should combine all hubs - opt-in by default, such that peers negotiate their installed mods and automatically (de)activate based on mod presence - self-describing message encoding that can be provided at runtime - ideally we would have tooling that would allow for a compilation phase during mod installation - in liue of that, it needs to be a format passed from the mod at runtime - preferably some string type like SJSON that can be parsed easily at runtime, but could also be a custom format based on Lua tables Implementation details: - https://libp2p.io/

lucas added the

stage

design

label 2025-05-04 16:06:50 +02:00

lucas pinned this 2025-05-04 16:06:59 +02:00

lucas commented

2025-05-08 15:45:28 +02:00

Author

Owner

It might be necessary to evaluate the performance of having a loose, self-describing format like SJSON vs something like Stingray's strict, pre-defined .network_config.
The latter can significantly decrease the amount of data transferred (especially by dropping field names), while the former has greater flexibility by allowing arbitrary tables, makes backwards compatibility possible, and doesn't require a complicated syntax to define data types.

I guess the best approach would even be to make both possible. Give create_* a type parameter where one of the two options can be chosen.

Overall API:

p2p.create_rpc(name, opts): Create a new RPC with the given name.
- name must not contain a dot, so that we can use the dot as separator (unless something in libp2p makes a different separator useful)
  - / might be a good option too, since the nature of directories makes it impossible for that to be in a mod name
- opts = { type, namespace, args }:
  - type denotes the type above, defaults to the loose one
  - namespace defaults to the mod's name (with string replacements to make it a suitable modifier, like lowercase and _ for space)
  - args to provide the argument definitions
- Recognize when the same name is used for incompatible definitions (e.g. type or args don't match)
p2p.send_rpc(name, args): Call an RPC
- name: if no namespace given, use the mod's own namespace. See remarks about separator above
- args a table of arguments, either arbitrary, or validated, depending on RPC type
- Will likely needs a parameter, or several variants for
  - send to a specific peer
  - send to everyone but me
  - send to everyone including me (technically just a QoL, though, since one could always call the callback themselves)
p2p.receive_rpc(name, callback): Register a callback for the RPC. See p2p.send_rpc for name parameter
- callback will receive a single args table. If something else is needed, it needs to be curried
p2p.create_object(name, opts): Create a new, shared object
- Returns a table with __index and __newindex metatables setup up such that the fields are mapped to a synchronized storage
- The table should also provide some methods to listen for changes
- name: See p2p.create_rpc
- opts: See p2p.create_rpc
- If type is loose, provide methods to create fields at runtime. __newindex might be enough, if not some kind of register_field.
- Provide methods to check whether an object has a mirror, and on which peer(s).
  - This needs to exist separately, even if there is an overall way to check if there are peers for a mod, since different versions of a mod may not register the same objects.
p2p.fetch_object(name): Create a shared object, but expect another client to define it
- I'm not sure if there is actually a use case for this, it's mostly just an idea for now
- The idea being that it might be useful to get a shared object, but letting another entity define its properties
  - e.g. one mod might want to use another mod's shared object, and not having to copy the exact definition makes incompatibilities less likely
- This could also become complex to implement, if there needs to be a system to delay this registration

Thoughts about versioning:

It may become useful to encourage versioning for RPCs/objects.
For individual entries, that's probably relatively simple, by just encouraging people to create _v2, _v3 suffixes, and then each mod can decide themselves if they want to implement backwards compatibility.
Enforcing the scheme in code will likely be hard, unless I also force people to make add a _v1, but we could at least encourage in through documentation.

Additionally, it might be useful in the long run, if the library can recognize when clients run different versions of a mod. Comparing their calls to p2p.create_* would be a way to realize this implicitly.
But it could also be beneficial to the ecosystem as a whole, if we just force mods to provide a mod.version from the start, even if we don't need it right now.

And to avoid the plethora of versioning schemes, that field should probably be restricted to a reasonable set like

single, incrementing integer
date string in \d{4}-\d{2}-\d{2}
SemVer-like \d+\.\d+\.\d+

It might be necessary to evaluate the performance of having a loose, self-describing format like SJSON vs something like Stingray's strict, pre-defined `.network_config`. The latter can significantly decrease the amount of data transferred (especially by dropping field names), while the former has greater flexibility by allowing arbitrary tables, makes backwards compatibility possible, and doesn't require a complicated syntax to define data types. I guess the best approach would even be to make both possible. Give `create_*` a `type` parameter where one of the two options can be chosen. Overall API: - `p2p.create_rpc(name, opts)`: Create a new RPC with the given name. - `name` must not contain a dot, so that we can use the dot as separator (unless something in libp2p makes a different separator useful) - `/` might be a good option too, since the nature of directories makes it impossible for that to be in a mod name - `opts = { type, namespace, args }`: - `type` denotes the type above, defaults to the loose one - `namespace` defaults to the mod's name (with string replacements to make it a suitable modifier, like lowercase and `_` for space) - `args` to provide the argument definitions - Recognize when the same name is used for incompatible definitions (e.g. `type` or `args` don't match) - `p2p.send_rpc(name, args)`: Call an RPC - `name`: if no namespace given, use the mod's own namespace. See remarks about separator above - `args` a table of arguments, either arbitrary, or validated, depending on RPC type - Will likely needs a parameter, or several variants for - send to a specific peer - send to everyone but me - send to everyone including me (technically just a QoL, though, since one could always call the callback themselves) - `p2p.receive_rpc(name, callback)`: Register a callback for the RPC. See `p2p.send_rpc` for `name` parameter - `callback` will receive a single `args` table. If something else is needed, it needs to be curried - `p2p.create_object(name, opts)`: Create a new, shared object - Returns a table with `__index` and `__newindex` metatables setup up such that the fields are mapped to a synchronized storage - The table should also provide some methods to listen for changes - `name`: See `p2p.create_rpc` - `opts`: See `p2p.create_rpc` - If `type` is loose, provide methods to create fields at runtime. `__newindex` might be enough, if not some kind of `register_field`. - Provide methods to check whether an object has a mirror, and on which peer(s). - This needs to exist separately, even if there is an overall way to check if there are peers for a mod, since different versions of a mod may not register the same objects. - `p2p.fetch_object(name)`: Create a shared object, but expect another client to define it - I'm not sure if there is actually a use case for this, it's mostly just an idea for now - The idea being that it might be useful to get a shared object, but letting another entity define its properties - e.g. one mod might want to use another mod's shared object, and not having to copy the exact definition makes incompatibilities less likely - This could also become complex to implement, if there needs to be a system to delay this registration --- Thoughts about versioning: It may become useful to encourage versioning for RPCs/objects. For individual entries, that's probably relatively simple, by just encouraging people to create `_v2`, `_v3` suffixes, and then each mod can decide themselves if they want to implement backwards compatibility. Enforcing the scheme in code will likely be hard, unless I also force people to make add a `_v1`, but we could at least encourage in through documentation. Additionally, it might be useful in the long run, if the library can recognize when clients run different versions of a mod. Comparing their calls to `p2p.create_*` would be a way to realize this implicitly. But it could also be beneficial to the ecosystem as a whole, if we just force mods to provide a `mod.version` from the start, even if we don't need it right now. And to avoid the plethora of versioning schemes, that field should probably be restricted to a reasonable set like - single, incrementing integer - date string in `\d{4}-\d{2}-\d{2}` - SemVer-like `\d+\.\d+\.\d+`

lucas commented

2025-05-08 16:32:50 +02:00

Author

Owner

local mod = get_mod("<self>")
local p2p = P2P.new(mod)

local game_object = p2p:create_object("foo")

I don't like the opposite way that some people have been doing:

local mod = get_mod("<self>")
P2P.register(mod)

local game_object = mod:create_object("foo")

This pollutes the mod object, is prone to run into issues about method names, and it makes it unclear where a function comes from.

A slightly better way would be to at least prefix the method, e.g. mod.p2p_create_object.
I guess that would actually have the benefit of not requiring local p2p = P2P.create(mod) or local p2p = mod.p2p in every file.

For comparison, the third option would be

local mod = get_mod("<self>")

local game_object = P2P.create_object(mod, "foo")

Actually, seeing it laid out like that, I do like the last option, actually.

It allows using the global P2P, which makes it both easier for the user, since they don't have to find a way to shuffle their instance around, and for the plugin implementation, since we only need lua.add_module_function, and no nested thing creating libraries.
It is a simpler concept than "first create an instance, then call methods on that"

But it does require the functions to be mostly stateless, since there is no instance to pass around.
Though I guess an entry the Lua registry, keyed by the mod would be an option.

Since the namespaces, and maybe the versioning, require knowing stuff about the particular mod that is going to call these functions, it would probably make sense to have one intermediate step that registers the mod once, instead of requiring it as argument in every function call: ```lua local mod = get_mod("<self>") local p2p = P2P.new(mod) local game_object = p2p:create_object("foo") ``` I don't like the opposite way that some people have been doing: ```lua local mod = get_mod("<self>") P2P.register(mod) local game_object = mod:create_object("foo") ``` This pollutes the `mod` object, is prone to run into issues about method names, and it makes it unclear where a function comes from. A slightly better way would be to at least prefix the method, e.g. `mod.p2p_create_object`. I guess that would actually have the benefit of not requiring `local p2p = P2P.create(mod)` or `local p2p = mod.p2p` in every file. For comparison, the third option would be ```lua local mod = get_mod("<self>") local game_object = P2P.create_object(mod, "foo") ``` --- Actually, seeing it laid out like that, I do like the last option, actually. - It allows using the global `P2P`, which makes it both easier for the user, since they don't have to find a way to shuffle their instance around, and for the plugin implementation, since we only need `lua.add_module_function`, and no nested thing creating libraries. - It is a simpler concept than "first create an instance, then call methods on that" But it does require the functions to be mostly stateless, since there is no instance to pass around. Though I guess an entry the Lua registry, keyed by the mod would be an option.

lucas added this to the v0.1 milestone 2025-05-09 00:43:04 +02:00

lucas commented

2025-05-09 00:52:15 +02:00

Author

Owner

Something that's still unclear: Is there a need for a host/client model, or can all peers be equal?

Probably not in a way that the p2p library forces it for all mods. If certain mods need this model, they should be able to negotiate it by themselves.

Though I guess if it were forced by the library, that could be helpful insofar that it's guaranteed to be the same peer for all mods.
It also depends on how much effort it actually is to implement that negotiation. If it is non-trivial, moving it into the library would also save on mods having to re-invent the wheel (and most of them probably poorly).

And I guess there is a certain range in how much the library implements. It could also be as simple as providing a function "which peer was the first in this session?" (which is a question that can be answered even if multiple peers came and left already). Though keeping track of join times only for this wouldn't make sense, either.

However, one thing that could make a host/client model very hard to implement is the fact that in Fatshark's sessions the dedicated server is the host, who will never leave, and so it might actually be possible that within a single mission, all original players leave and the session stays active with the newer players. If that situation is possible, then host migration would be mandatory, and that's quite the beast to implement.
That would actually make a good argument to not provide any concept of a "host" in the library, and to also discourage mods from relying too much on it.

Something that's still unclear: Is there a need for a host/client model, or can all peers be equal? Probably not in a way that the p2p library forces it for all mods. If certain mods need this model, they should be able to negotiate it by themselves. Though I guess if it were forced by the library, that could be helpful insofar that it's guaranteed to be the same peer for all mods. It also depends on how much effort it actually is to implement that negotiation. If it is non-trivial, moving it into the library would also save on mods having to re-invent the wheel (and most of them probably poorly). And I guess there is a certain range in how much the library implements. It could also be as simple as providing a function "which peer was the first in this session?" (which is a question that can be answered even if multiple peers came and left already). Though keeping track of join times only for this wouldn't make sense, either. However, one thing that could make a host/client model very hard to implement is the fact that in Fatshark's sessions the dedicated server is the host, who will never leave, and so it might actually be possible that within a single mission, all original players leave and the session stays active with the newer players. If that situation is possible, then host migration would be mandatory, and that's quite the beast to implement. That would actually make a good argument to not provide any concept of a "host" in the library, and to also discourage mods from relying too much on it.

lucas added the

kind

discussion

label 2025-05-09 01:01:54 +02:00

lucas referenced this issue

2025-05-09 01:32:51 +02:00

Collection of potential use cases #6