Camobiwon•2mo ago

Best way for syncing data "generically" in Normcore?

I have a strange use case where I basically want to reuse the same model type but allow different types of data. The best way I can think of off the top of my head is via byte[], however I'm unsure if this is ideal and additionally I'm curious what methods you guys may be using to quickly covert to and from a byte array?

9 Replies

kapa.ai•2mo ago

Hi @Camobiwon. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave: ---------------------- Based on the knowledge sources provided, I can offer some insights on syncing data "generically" in Normcore, particularly regarding the use of byte[] arrays. Normcore does support byte[] arrays as properties in RealtimeModels. However, there are some important considerations to keep in mind: 1. Equality Checks: By default, Normcore only checks the reference of the byte**] array, not its contents. This means updates will only be synchronized if you set a new reference to the array. To address this, you can disable equality checks for the property [as described in the documentation (https://normcore.io/documentation/room/supported-primitives#support-for-byte-arrays):
[RealtimeProperty(1, false, true, includeEqualityCheck: false)] private byte[] _rawData = new byte[0];
[RealtimeProperty(1, false, true, includeEqualityCheck: false)] private byte[] _rawData = new byte[0];
2. Updating the property: When you modify the contents of the byte[] array, you need to explicitly update the property to trigger synchronization:
// Update a byte in the buffer rawData[0] = 0xff; // Update the property to tell Normcore the value has changed model.rawData = rawData;
// Update a byte in the buffer rawData[0] = 0xff; // Update the property to tell Normcore the value has changed model.rawData = rawData;
As for quickly converting to and from a byte array, the documentation doesn't provide specific methods for this. You would need to implement your own serialization and deserialization methods based on your specific data types. It's worth noting that while `byte]` can be used for generic data storage, it may not be the most efficient or convenient approach for all scenarios. Normcore provides other options for storing collections of data, such as [RealtimeArray, RealtimeSet, and RealtimeDictionary (https://normcore.io/documentation/room/collections), which might be more suitable depending on your specific use case. If you need to store large amounts of data or frequently changing data, keep in mind that datastores are limited to 10mb, and a single datastore update is limited to 125kb** (https://normcore.io/documentation/room/common-questions#is-there-a-size-limit). For larger data, it's recommended to store it externally (e.g., in S3 or Google Cloud Storage) and only keep references in the Normcore datastore. Given the limited information about your specific use case, it's difficult to recommend the best approach. If you need more tailored advice, you might want to provide more details about the types of data you're working with and your synchronization requirements. ---------------------- Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: ----------------------

kapa.ai•2mo ago

<@271788139381653514> Kapa.ai is still learning and improving, please let me know how I did by reacting below.

NormalMark•2mo ago

I'm curious about your use case, could you describe it a bit more? Is the "genericness" known at compile-time, or is it runtime? Ex would this hypothetical API be suitable for your use case:

c#
[RealtimeModel]
public partial class TestModel<T> {
    [RealtimeProperty(1, true)]
    private T _test;
}

c#
[RealtimeModel]
public partial class TestModel<T> {
    [RealtimeProperty(1, true)]
    private T _test;
}

CamobiwonOP•2mo ago

It's a model used in a RealtimeSet, multiple things can call this and use the byte array genetically, could be say a float, int, vector3, etc.. Scripts already need to know the type to serialize / deserialize so it would be known at compile time, the API seems like it'd work quite well but I do also have some additional data besides just the generic one Looked into this from 2.13 and sadly it doesn't seem like it works for my case However I am glad that it was added at all! I'm sure I'll have a use case in the future

NormalMark•2mo ago

Hey! I'm working on a Normcore update that adds support for the hypothetical API described above. It's close to done but I'll be out of office for a week and I'll only be able to finish it when I come back. If you need a workaround in the meantime you can try nesting models. This can be ex by adding a model to a realtime collection (and using inheritance instead of generics). It's certainly less than ideal. There's probably other creative solutions that I can't see atm. But built-in support for generics definitely beats them all. Ex (inheritance instead of generics):

c#
[RealtimeModel]
MyBaseModel { ... }

[RealtimeModel]
MyIntModel : MyBaseModel { ... }

c#
[RealtimeModel]
MyBaseModel { ... }

[RealtimeModel]
MyIntModel : MyBaseModel { ... }

Anthony•6d ago

Is this still hypothetical? I'd love to be able to use generics in this way but I'm getting an invalid property type error (v2.13.2)

NormalMark•6d ago

Still hypothetical but I'm curious about your use case The difficult part in the implementation is that if T derives from RealtimeModel then our source generator generates different code from if it's any other type (like int, float, etc). Basically ReliableProperty/UnreliableProperty vs ModelProperty code inside the auto-generated model. So it's like we could support a generic TestModel<T> but the user needs to also declare if T is a model or not. Based on which we generate different code, but that would be very clunky. Because of this I decided to hold off on shipping the prototype.

Anthony•5d ago

Gotcha, makes sense! In my particular case, I have a ton of key/value sets of all different data types where their handler classes all have the same code, so I'd ideally want to do something like:

public class ValueSetHandler<T> : RealtimeComponent<ValueDictionaryModel<T>>
    {
        /// etc..
    }
    
    [RealtimeModel]
    public partial class ValueDictionaryModel<T>
    {
        [RealtimeProperty(1, true, true)]
        private StringKeyDictionary<ValueModel<T>> _dictionary;
    }
    
    [RealtimeModel]
    public partial class ValueModel<T>
    {
        [RealtimeProperty(1, true, true)]
        private T _value;
    }

public class ValueSetHandler<T> : RealtimeComponent<ValueDictionaryModel<T>>
    {
        /// etc..
    }
    
    [RealtimeModel]
    public partial class ValueDictionaryModel<T>
    {
        [RealtimeProperty(1, true, true)]
        private StringKeyDictionary<ValueModel<T>> _dictionary;
    }
    
    [RealtimeModel]
    public partial class ValueModel<T>
    {
        [RealtimeProperty(1, true, true)]
        private T _value;
    }

Granted, what I'm doing probably isn't terribly common.. I've got a framework that hides all 3rd party network code/libs behind an abstraction layer so it can be removed or changed at any time

NormalMark•5d ago

Ah I understand now. Your best best atm might be to try this:

c#
[RealtimeModel]
public partial class ValueDictionaryModel<TValueModel> where TValueModel : RealtimeModel
{
    [RealtimeProperty(1, true, true)]
    private StringKeyDictionary<TValueModel> _dictionary;
}

interface IValueModel<T>
{
    public value { get; set; }
}

[RealtimeModel]
public partial class IntModel : IValueModel<T>
{
    [RealtimeProperty(1, true)]
    private int _value;
    
    // IValueModel<T>.value is implemented by the auto-generated model code
}

c#
[RealtimeModel]
public partial class ValueDictionaryModel<TValueModel> where TValueModel : RealtimeModel
{
    [RealtimeProperty(1, true, true)]
    private StringKeyDictionary<TValueModel> _dictionary;
}

interface IValueModel<T>
{
    public value { get; set; }
}

[RealtimeModel]
public partial class IntModel : IValueModel<T>
{
    [RealtimeProperty(1, true)]
    private int _value;
    
    // IValueModel<T>.value is implemented by the auto-generated model code
}

And so on... creating specific models for each primitive type you need to support. The full list of types we support in Normcore is: https://docs.normcore.io/room/supported-primitives

Best way for syncing data "generically" in Normcore?

Did you find this page helpful?