Deserialization in Unity using Converters

[zory]

Hello,
Probably I'm stuck in obvious case, because I can't find any documentation on this.. Maybe you could provide some example?
I'm trying to serialize/deserialize objects which contains List

I'm getting
ArgumentException: Could not cast or convert from System.String to UnityEngine.Vector3Int.

My code:

private static JsonSerializerSettings settings = new JsonSerializerSettings()
{
    ContractResolver = new Newtonsoft.Json.UnityConverters.UnityTypeContractResolver(),
    Converters = new JsonConverter[] {
        new Newtonsoft.Json.UnityConverters.Math.Vector3IntConverter(),
    }
};
//--- and the serialization/deserialization
var strVal = JsonConvert.SerializeObject(instance, settings);
var instance = JsonConvert.DeserializeObject<T>(strVal, settings);

Thanks in advance.

[applejag]

Hi @zory! Thanks for reporting this

Though I'm having troubles understanding the full picture. Will try to reproduce it myself and hopefully find some issue.

Could you post the full stacktrace of the error?

[zory]

This is full stack trace. One of my structures GenericDictionary have its own ISerializationCallbackReceiver implementation. I will investigate, maybe it is breaking things.

ArgumentException: Could not cast or convert from System.String to UnityEngine.Vector3Int.
Newtonsoft.Json.Utilities.ConvertUtils.EnsureTypeAssignable (System.Object value, System.Type initialType, System.Type targetType) (at /root/repo/Src/Newtonsoft.Json/Utilities/ConvertUtils.cs:624)
Newtonsoft.Json.Utilities.ConvertUtils.ConvertOrCast (System.Object initialValue, System.Globalization.CultureInfo culture, System.Type targetType) (at /root/repo/Src/Newtonsoft.Json/Utilities/ConvertUtils.cs:595)
Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EnsureType (Newtonsoft.Json.JsonReader reader, System.Object value, System.Globalization.CultureInfo culture, Newtonsoft.Json.Serialization.JsonContract contract, System.Type targetType) (at /root/repo/Src/Newtonsoft.Json/Serialization/JsonSerializerInternalReader.cs:988)
Rethrow as JsonSerializationException: Error converting value "(2, 0, -2)" to type 'UnityEngine.Vector3Int'. Path 'elevationMap['(2, 0, -2)']', line 1, position 82.
Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EnsureType (Newtonsoft.Json.JsonReader reader, System.Object value, System.Globalization.CultureInfo culture, Newtonsoft.Json.Serialization.JsonContract contract, System.Type targetType) (at /root/repo/Src/Newtonsoft.Json/Serialization/JsonSerializerInternalReader.cs:992)
Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateDictionary (System.Collections.IDictionary dictionary, Newtonsoft.Json.JsonReader reader, Newtonsoft.Json.Serialization.JsonDictionaryContract contract, Newtonsoft.Json.Serialization.JsonProperty containerProperty, System.String id) (at /root/repo/Src/Newtonsoft.Json/Serialization/JsonSerializerInternalReader.cs:1404)
Rethrow as JsonSerializationException: Could not convert string '(2, 0, -2)' to dictionary key type 'UnityEngine.Vector3Int'. Create a TypeConverter to convert from the string to the key type object. Path 'elevationMap['(2, 0, -2)']', line 1, position 82.
Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateDictionary (System.Collections.IDictionary dictionary, Newtonsoft.Json.JsonReader reader, Newtonsoft.Json.Serialization.JsonDictionaryContract contract, Newtonsoft.Json.Serialization.JsonProperty containerProperty, System.String id) (at /root/repo/Src/Newtonsoft.Json/Serialization/JsonSerializerInternalReader.cs:1438)
Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) (at /root/repo/Src/Newtonsoft.Json/Serialization/JsonSerializerInternalReader.cs:568)
Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) (at /root/repo/Src/Newtonsoft.Json/Serialization/JsonSerializerInternalReader.cs:300)
Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue (Newtonsoft.Json.Serialization.JsonProperty property, Newtonsoft.Json.JsonConverter propertyConverter, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerProperty, Newtonsoft.Json.JsonReader reader, System.Object target) (at /root/repo/Src/Newtonsoft.Json/Serialization/JsonSerializerInternalReader.cs:1039)
Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject (System.Object newObject, Newtonsoft.Json.JsonReader reader, Newtonsoft.Json.Serialization.JsonObjectContract contract, Newtonsoft.Json.Serialization.JsonProperty member, System.String id) (at /root/repo/Src/Newtonsoft.Json/Serialization/JsonSerializerInternalReader.cs:2415)
Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) (at /root/repo/Src/Newtonsoft.Json/Serialization/JsonSerializerInternalReader.cs:493)
Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) (at /root/repo/Src/Newtonsoft.Json/Serialization/JsonSerializerInternalReader.cs:300)
Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize (Newtonsoft.Json.JsonReader reader, System.Type objectType, System.Boolean checkAdditionalContent) (at /root/repo/Src/Newtonsoft.Json/Serialization/JsonSerializerInternalReader.cs:202)
Newtonsoft.Json.JsonSerializer.DeserializeInternal (Newtonsoft.Json.JsonReader reader, System.Type objectType) (at /root/repo/Src/Newtonsoft.Json/JsonSerializer.cs:907)
Newtonsoft.Json.JsonSerializer.Deserialize (Newtonsoft.Json.JsonReader reader, System.Type objectType) (at /root/repo/Src/Newtonsoft.Json/JsonSerializer.cs:886)
Newtonsoft.Json.JsonConvert.DeserializeObject (System.String value, System.Type type, Newtonsoft.Json.JsonSerializerSettings settings) (at /root/repo/Src/Newtonsoft.Json/JsonConvert.cs:837)
Newtonsoft.Json.JsonConvert.DeserializeObject[T] (System.String value, Newtonsoft.Json.JsonSerializerSettings settings) (at /root/repo/Src/Newtonsoft.Json/JsonConvert.cs:792)
Logic.Multiplayer.CommunicationLayer+<>c__2`1[T].<RegisterGenericSerializationHandler>b__2_1 (System.IO.Stream stream) (at Assets/Scripts/Logic/Both/Multiplayer/CommunicationLayer.cs:57)
MLAPI.Serialization.SerializationManager+<>c__DisplayClass7_0`1[T].<RegisterSerializationHandlers>b__1 (System.IO.Stream stream) (at <03461c87e6064468b4284725fde36c13>:0)
MLAPI.Serialization.SerializationManager.TryDeserialize (System.IO.Stream stream, System.Type type, System.Object& obj) (at <03461c87e6064468b4284725fde36c13>:0)
MLAPI.Serialization.BitReader.ReadObjectPacked (System.Type type) (at <03461c87e6064468b4284725fde36c13>:0)
MLAPI.Messaging.ReflectionMethod.InvokeReflected (MLAPI.NetworkedBehaviour instance, System.IO.Stream stream) (at <03461c87e6064468b4284725fde36c13>:0)
MLAPI.Messaging.ReflectionMethod.Invoke (MLAPI.NetworkedBehaviour target, System.UInt64 senderClientId, System.IO.Stream stream) (at <03461c87e6064468b4284725fde36c13>:0)
MLAPI.NetworkedBehaviour.InvokeServerRPCLocal (System.UInt64 hash, System.UInt64 senderClientId, System.IO.Stream stream) (at <03461c87e6064468b4284725fde36c13>:0)
MLAPI.NetworkedBehaviour.SendServerRPCPerformance (System.UInt64 hash, System.IO.Stream messageStream, System.String channel, MLAPI.Security.SecuritySendFlags security) (at <03461c87e6064468b4284725fde36c13>:0)
MLAPI.NetworkedBehaviour.SendServerRPCBoxed (System.UInt64 hash, System.String channel, MLAPI.Security.SecuritySendFlags security, System.Object[] parameters) (at <03461c87e6064468b4284725fde36c13>:0)
MLAPI.NetworkedBehaviour.InvokeServerRpc[T1] (MLAPI.NetworkedBehaviour+RpcMethod`1[T1] method, T1 t1, System.String channel, MLAPI.Security.SecuritySendFlags security) (at <03461c87e6064468b4284725fde36c13>:0)
Logic.Multiplayer.Server.CreateMap (Logic.Map.MapInfo mapInfo) (at Assets/Scripts/Logic/Both/Multiplayer/Server.cs:40)
Logic.GlobalAccess.GameManager.CreateMap (Logic.Map.MapInfo mapInfo, System.Collections.Generic.List`1[T] characterMasterInfos, System.Collections.Generic.List`1[T] abilityInfos, System.Collections.Generic.List`1[T] tileObjectsInfos) (at Assets/Scripts/Logic/Both/Global/GameManager.cs:89)
Logic.GlobalAccess.GameManager.StartGameCoroutine () (at Assets/Scripts/Logic/Both/Global/GameManager.cs:80)
Logic.GlobalAccess.GameManager.StartGame () (at Assets/Scripts/Logic/Both/Global/GameManager.cs:74)
Logic.Multiplayer.MultiplayerManager.OnServerStarted () (at Assets/Scripts/Logic/Both/Multiplayer/MultiplayerManager.cs:73)
MLAPI.NetworkingManager.StartHost (System.Nullable`1[T] position, System.Nullable`1[T] rotation, System.Nullable`1[T] createPlayerObject, System.Nullable`1[T] prefabHash, System.IO.Stream payloadStream) (at <03461c87e6064468b4284725fde36c13>:0)
Logic.Multiplayer.MultiplayerManager.StartNewGame () (at Assets/Scripts/Logic/Both/Multiplayer/MultiplayerManager.cs:46)
Logic.GlobalAccess.GameManager.Start () (at Assets/Scripts/Logic/Both/Global/GameManager.cs:54)

[applejag]

Well something I can tell you right now is that the UnityTypeContractResolver type does currently not handle ISerializationCallbackReceivers. If you're depending on that then that's probably the reason this doesn't work.

It can be a future feature, but it's not available right now

[zory]

I tried removing everything, leaving just simple types, so it doesn't rely on ISerializationCallbackReceiver anymore.
Dictionary<Vector3Int, int> -> still Fails. But Vector3Int or List<Vector3Int> is working as expected. Moreover, Dictionary<int, int> works fine too.

[applejag]

Huh I haven't actually tested the converters to be used as Dictionary keys before. Have not researched how Newtonsoft.Json handles those situations. All I remember though is that it has special built-in cases for IEnumerable<T>, IList<T>, and IDictionary<TKey, TValue>, and some others, but I don't remember the details.

As a workaround you maybe could use the type IEnumerable<KeyValuePair<Vector3Int, int>> inside the class you're serializing, as Dictionaries inherit from that type so just do a little bit of polymorphism and I believe there's a constructor on Dictionary to turn it back into a dictionary. All just to force Newtonsoft.Json to serialize it with the regular IEnumerable<T> implementation. If that doesn't work either I'd be surprised.

[zory]

Yes, your workaround works. Serializing for example List<KeyValuePair<Vector3Int, int>> yields expected results. Thank you for letting me use your brain : ). If you would come up with another solution, let me know : ).

[MrFJ]

Hey, I just ran into a similar issue, though your solution does not seem to work for me. This:

public IEnumerable<KeyValuePair<Vector3Int, string>> EntitiesSerialized { get; set; } = new Dictionary<Vector3Int, string>();

is serialized into this:

"EntitiesSerialized": {
    "(4, 14, 0)": "PassiveEnemyShip",
    "(3, 11, 0)": "PassiveEnemyShip",
    "(1, 9, 0)": "PassiveEnemyShip"
}

Which JsonConvert refuses to deserialize. Any thoughts? I even tried adding the Vector3IntConverter manually to the serialize and deserialize calls.

[applejag]

Hi @MrFJ ! You're using a complex type as a dictionary key. That doesn't work in JSON, and Newtonsoft.JSON's solution for when doing this seems to just be to .ToString() the value. Parsing that is not an option.

You do save the type as IEnumerable<KeyValuePair<Vector3Int, string>>, but the underlying type that Newtonsoft.JSON sees is still the Dictionary<,>

Maybe you could try use an explicit list instead?

public List<KeyValuePair<Vector3Int, string>> EntitiesSerialized { get; set; } = new Dictionary<Vector3Int, string>().ToList();

[MrFJ]

Thanks for the quick response! I changed my implementation to:

public Dictionary<Vector3Int, string> Entities { get; set; } = new ();
public IEnumerable<KeyValuePair<Vector3Int, string>> EntitiesSerialized {
    get => Entities;
    set => Entities = (Dictionary<Vector3Int, string>)value;
}

to no avail, but it turns out ToString() was indeed the missing piece!

This works perfectly:

[JsonIgnore]
public Dictionary<Vector3Int, string> Entities { get; set; } = new ();

public IEnumerable<KeyValuePair<Vector3Int, string>> EntitiesSerialized {
    get => Entities.ToList();
    set => Entities = (Dictionary<Vector3Int, string>)value;
}

[applejag]

Great! Happy you found a solution :)

[MrFJ]

It seems I spoke too soon. This is my JSON:

"EntitiesSerialized": [
  {
    "Key": {
      "x": 6,
      "y": 12,
      "z": 0
    },
    "Value": "PassiveEnemyShip"
  },
  {
    "Key": {
      "x": 6,
      "y": 11,
      "z": 0
    },
    "Value": "PassiveEnemyShip"
  },
  {
    "Key": {
      "x": 6,
      "y": 8,
      "z": 0
    },
    "Value": "PassiveEnemyShip"
  }
]

The dictionary is empty after deserialization though :/ I tried changing IEnumerable to List, I even tried the following to no results:

private IEnumerable<KeyValuePair<Vector3Int, string>> entities;
public IEnumerable<KeyValuePair<Vector3Int, string>> EntitiesSerialized {
    get => Entities.ToList();
    set => entities = value;
}

I'm getting nada :(

[applejag]

It's getting complicated with the types here. To simplify, you could have it more explicit. Such as by using one class for serializing/deserializing, and one that you're working with otherwise.

Such pattern is usually called a DTO (Data-Transfer-Object).

So that before you serialize it, you always convert the dictionary into a List<KeyValuePair<,>>, and then after deserializing you always convert it back into a dictionary. You have then two classes, one that you use during runtime (such as a ScriptableObject or GameObject), and then as soon as you're serializing/deserializing you create this DTO and copy over the data to/from it.

Example:

https://dotnetfiddle.net/W4gUHS