Loading/Saving is inconsistently slow at specific times, when using encryption

[Benji23245]

Hello everyone,

I have a custom save/load system in Unity using JSON + a simple XOR encryption.
The code works perfectly in debug mode (encryption disabled), but in non‑debug mode the entire unity editor freezes for ~9 seconds only on the first Load() call after a scene change.

• Trying to print some text before the Load() call, doesn’t display it until after the freeze is over.
• If the scene was directly loaded from the editor (and not reached via gameplay), it’s always super fast.
• That issue only seems to occurs on the scenes before reaching the main menu for some unknown reason, as it is always super fast there, and also in action stages, hub worlds etc too.
• I made it so the saved file was smaller, to see if it had any impact, and surprisingly, up to 59 kb it took ~1 sec for the first Load instead of 9 (and then super fast). At 61 kb, it took the 9 secs (usual size is 65 kb).

After that first freeze, all subsequent loads in the same scene are instant.

Here’s my encryption function:

private static string EncryptDecrypt(string data)
{
    var sb = new System.Text.StringBuilder(data.Length);

    for (int i = 0; i < data.Length; i++)
    {
        sb.Append((char)(data[i] ^ encryptionCodeWord[i % encryptionCodeWord.Length]));
    }

    return sb.ToString();
}

If I don’t use it, it’s always super fast.

I’m on Windows 11. And since I write to files, I’m wondering if it could be some external thing unrelated to Unity, that’s blocking the thread.

Would anyone have an idea ?

Thanks!

[DevDunk]

I’d check with the profiler

[Kurt-Dekker]

XOR-ing 61kb of data should never be an issue speed-wise.

What is encryptionCodeWord, just an array? If it is actually an IList<T>-type of interface into some kind of byte sequence generator, who knows what that might be doing, particularly the .Length property.

As DevDunk suggests, profiler.

[Benji23245]

Hello,

encryptionCodeWord is just a plain string: “TYlcOS5CAn0q0Zg0RoWeKkW07WNgOqrE”

I’ll look at the profiling.

Thanks.

[tleylan]

I’ll defer to the others for your next step but why guess about how long it takes and under what circumstances?

Debug.Log could report the data parameter, data.length, the length of the result and of course the time it takes.

[Benji23245]

I tried using the profiler, but it showed this error in the console, seconds after I reached the freeze:

The system is running out of memory. Please close applications to free memory or free up space on the partition where the pagefile is located.
Used memory (physical and paged): 93% (25004MB of 26841MB).
Discarding profiler frames data.

EDIT: After closing every other app, I managed to have it work, however, I couldn’t see the information I needed.

The only places where I found my method, it was using very little %.
Also, I don’t see a long lasting spike or anything like that, and the profiler timeline stops entirely during the freeze, as if nothing happened at all within unity during the freeze.

EDIT2: I finally found it (I can see about 9000 ms, so 9 secs):

But I have no idea what to make of it…

[Benji23245]

By the way this is the whole FIleDataHandler script:

using UnityEngine;
using System;
using System.IO;
using Newtonsoft.Json;

internal static class FileDataHandler {

  internal static string dataDirPath = Application.persistentDataPath;
  internal const string dataFileName = "SA3_save_";
  internal const string debug = "debug_";
  internal const string fileExtension = ".json";
  internal const int maxSaveIndex = 12;

  private static readonly string encryptionCodeWord = "TYlcOS5CAn0q0Zg0RoWeKkW07WNgOqrE";

  internal static GameData Load(int index) {
    string fullPath = System.IO.Path.Combine(dataDirPath, $"{dataFileName}{(GameManager.isDebug ? debug : "")}{index}{fileExtension}");
    GameData loadedData = null;

    if (File.Exists(fullPath)) {
      try {
        string dataToLoad = "";

        using (FileStream stream = new(fullPath, FileMode.Open)) {
          using (StreamReader reader = new(stream)) {
            dataToLoad = GameManager.isDebug ? reader.ReadToEnd() : EncryptDecrypt(reader.ReadToEnd());
          }
        }

        JsonSerializerSettings settings = new() { ObjectCreationHandling = ObjectCreationHandling.Replace, TypeNameHandling = TypeNameHandling.Auto, NullValueHandling = NullValueHandling.Ignore };
        loadedData = JsonConvert.DeserializeObject<GameData>(dataToLoad, settings);
      }
      catch (Exception e) {
        Debug.LogError("An error occured when trying to load data from file: " + fullPath + "\n" + e);
      }
    }

    return loadedData;
  }

  internal static void Save(GameData data) {
    string fullPath = System.IO.Path.Combine(dataDirPath, $"{dataFileName}{(GameManager.isDebug ? debug : "")}{GameManager.currentSaveIndex}{fileExtension}");

    try {
      Directory.CreateDirectory(System.IO.Path.GetDirectoryName(fullPath));

      JsonSerializerSettings settings = new() { ReferenceLoopHandling = ReferenceLoopHandling.Ignore, TypeNameHandling = TypeNameHandling.Auto };
      string dataToSave = GameManager.isDebug ? JsonConvert.SerializeObject(data, Formatting.Indented, settings) : EncryptDecrypt(JsonConvert.SerializeObject(data, Formatting.Indented, settings));

      using (FileStream stream = new(fullPath, FileMode.Create)) {
        using (StreamWriter writer = new(stream)) {
          writer.Write(dataToSave);
        }
      }
    }
    catch (Exception e) {
      Debug.LogError("An error occured when trying to save data to file: " + fullPath + "\n" + e);
    }
  }

  internal static void Delete(int index) {
    string fullPath = System.IO.Path.Combine(dataDirPath, $"{dataFileName}{(GameManager.isDebug ? debug : "")}{index}{fileExtension}");

    if (File.Exists(fullPath)) {
      try {
        File.Delete(fullPath);
      }
      catch (Exception) {
        Debug.LogError("An error occured when trying to delete save file.");
      }
    }
    else Debug.LogError("File not found.");

    Debug.Log("File successfully deleted!");
  }

  private static string EncryptDecrypt(string data) {
    var sb = new System.Text.StringBuilder(data.Length);

    for (int i = 0; i < data.Length; i++) {
      sb.Append((char)(data[i] ^ encryptionCodeWord[i % encryptionCodeWord.Length]));
    }

    return sb.ToString();
  }

}

[Kurt-Dekker]

That looks like it is loading a giant piece of audio… 28k read operations and a large allocation.

I think the correlation between encrypted and not encrypted has nothing to do with it directly.

Delete your audio and retry your encrypted / not encrypted test.

[CodeSmile]

@Benji23245

a simple XOR encryption.

That’s not encryption at all.

Put your json into an agent and tell it to “decrypt” it. You’ll get it back in no time as plain text.

@Benji23245

• Trying to print some text before the Load() call, doesn’t display it until after the freeze is over.

Expected because this happens in the same frame, and the console window or editor.log probably too only update when the frame ends.

The most likely problem is whatever the loaded json does after it’s been loaded, ie during processing of the loaded data. And since you use Resources.LoadAll() you can’t really complain …

How many bytes do you have in Resources? If it’s hundreds of MB, those will get loaded all at once (and possibly need decompressing too). Resources.LoadAll is really a poor man’s resource management system. It should be reserved for very specific use-cases ie you have all your gamedata json files autogenerated and they’re just a bunch of Kb and get processed by a custom loader system when the game launches.

[tleylan]

It appears to be time to ask, “what is your goal in doing this”?

It may not accomplish your goal or there may be another way to accomplish your goal.

If you’re hoping for some security we know that XORing the data won’t really provide that. You might be able to speed the process by just eliminating the process.

[Kurt-Dekker]

@CodeSmile

That’s not encryption at all.

Remember that telling someone encryption doesn’t work usually makes them reply with “So you’re saying…” and then go into cognitive dissonance.

XOR “encryption” has several major advantages:

• it’s fast and easy to implement
• it’s fast and easy to execute
• since the dev is often unable to “crack” their own "encryption, they think they’re pulling one over on the 1337 haxors of the world.
• it fools the gamedev by making them think they solved a problem
• it allows them to move onto something that actually improves their game… WIN!

Incoming blurb:

If you’re concerned about the user ‘hacking your save files,’ or ‘cheating in your game,’ which is playing on their computer, just don’t be.

There’s nothing you can do about it. Nothing is secure, it is not your computer, it is the user’s computer.

If it must be secure from tampering, store it on your own server and have the user connect to download it. Remember they can still change it after they download it and before your program uses it.

If it must have secure user authentication, use Apple login or Google login. Nothing else is worth it.

Anything else is a waste of your time and the only person you’re going to inconvenience is yourself when you’re debugging the game and you have savegame errors. Work on your game instead.

Remember, it only takes one 12-year-old in Finland to write a script to read/write your game files and everybody else can now use that script. Read about Cheat Engine to see more ways you cannot possibly control this.

NOTE: if you begin your response with “So you’re saying that…”, then you haven’t read the above. Go back and read it carefully or you are going to waste a lot of your own development time for nothing.

Protect?! Against who, Ze Germans?!

Snatch, just Ze Germans

[Benji23245]

My goal is not to make military grade security, it’s just to reduce the amount of people who would cheat.

I know people can just download 100% save games or whatever, without even any hacking at all.

I feel like the majority would be more likely to cheat if they just had a json file they could edit in seconds, rather than if they had that « encryption » barrier that would discourage most of them.

Plus, why do most prod games do it then ? It’s a genuine question I want to understand, cause that’s the main reason I did it in the first place.

Thanks

[Kurt-Dekker]

@Benji23245

Plus, why do most prod games do it then ?

I suppose you should ask a developer who does this.

Who knows, maybe they’re only doing it because they see others trying to do it?

@Benji23245

it’s just to reduce the amount of people who would cheat.

And this impacts your pocketbook how?

Debugging and implementing any form of encryption DOES have a negative cost (time spent on it), so one would hope that having it in would have a POSITIVE value, but I have yet to see this.

[Bunny83]

@CodeSmile

That’s not encryption at all.

Well, while it’s not a very good one, it’s still an encryption, a quite common one. Also without any information about what the decrypted data may look like, it can take quite some efford to figure out a 33 length xor sequence. A lot of encryption methods have a simple xor at the final stage. The “best” encryption would be xoring your data with a non deterministic key that has the same length as the data. There’s absolutely no way to decrypt it unless you know the whole key. Of course having a small key and repeat it makes it attackable. Though as I said without knowing or at least having an idea about what the decrypted data actually is, it’s not that simple. Keep in mind we talk about a situation of an attacker who is not able to decompile or debug the code in action. So the basic assumption is that he doesn’t have any insight in what the data may be. If you assume a skilled cracker, no encryption will ever work since you have to ship the key with your application. It’s just a way to make it harder for the average script kiddy.

I personally would always just use a simple PRNG with either a known fix seed or a random seed and save the seed with the data. Xor with the random number stream and you’re good to go. It’s also not that secure, but much harder to crack with simple correlation. Since I started programming way back with Turbo Pascal, I quite like the simple LCG generator it shipped with. It’s literally just

int state;
int Next()
{
    return state = state * 0x8088405 + 1;
}

It has a 32 bit range (so no duplicate in the 32 bit range until it wraps around) and it’s really simple. Though Unity’s xorshift128 is of course even better. I once made an xorshift64 implementation. It’s also extremely simple.

@Benji23245

I feel like the majority would be more likely to cheat if they just had a json file they could edit in seconds, rather than if they had that « encryption » barrier that would discourage most of them.

Yes, I get that sentiment and it’s what many games do just to provide a little bit of drag. Though instead of an encryption, you can also use some kind of checksum and if it doesn’t match, just report a broken save or make it load, wait for 2 minutes and then report the broken save Though you should make clear that there’s an issue with a corrupted save so the players don’t get angry over loosing their save file. Though as others have said, if someone wants to cheat in a single player game, let them.

ps: My SimpleJSON has a binary extension which can serialize the json to a simple custom binary format. With the SharpZipLib you can also output a compressed file. It’s also not encrypted, but not that easy to edit for most users.
Your encryption, assuming the user / player knows the key from a forum, they don’t even need to know any programming. CyberChef has an xor node that allows you to enter a repeated key So it’s that common

[tleylan]

This topic has come up many times and there are tradeoffs. Clearly the XOR solution is at the extremely limited to have any meaningful impact end of the spectrum.

Is it a multiplayer app? If not is there any meaningful reason to stop a person from cheating? If it is popular app/game the decryption scheme will exist on a website, post the file and get the decrypted version as a response. Add a small editor to the webpage and they can just decrypt, edit and encrypt before downloading.

Is the settings file the only hackable entity? Can ILSpy or dotPeek determine the string constant used in encryptionCodeWord?

It doesn’t mean it isn’t a good idea to protect settings but this doesn’t do it.

[bugfinders]

why not just use securestring?

[tleylan]

Probably he doesn’t know what it is (like me) Do you use it? Does it handle your use case(s)?

To quote Brave AI, However, SecureString has significant limitations: it is Windows-only, deprecated by Microsoft for new development, and offers limited protection because the string must eventually be converted to a plain text string for most APIs to use it.

If the OP is “encrypting” a giant piece of audio leaving that unencrypted is probably workable. The core question to all the solutions is (as I see it) “what do you want/expect it to do”?

[spiney199]

Seems like OP calling Resources.LoadAll is causing Unity to load more than they’re expecting. Must be using the non-generic version which I believe just literally loads absolutely everything into memory all at once.

Not sure how your save data is structured, but probably sounds like some things need to use the generic LoadAll<T> method (or find a better way to load assets), or these assets should be lazily initialised so they aren’t performed as soon as they are instanced. It all seem to spur from your GameData constructor, and a constructor should not be causing so much work to be done.

[Kurt-Dekker]

@spiney199

Must be using the non-generic version which I believe just literally loads absolutely everything into memory all at once.

Resources.LoadAll<T>() is actually even worse than that: even supplying a T, it STILL loads every single asset at the given URI, then ransacks them for matching T and returns an array of only those T.

That’s why subdirectories are critical… once you get used to the bad assumptions baked into the Resources.Load* API, it really is pretty handy if you’re not keen on spending a lot of time organizing your assets (that’s me!).

Using the <T> version is critical for other reasons too:

Here’s my standard blurb-a-roonie:

Always use Resources.Load<T>(), never use Resources.Load() as T

Resources.Load not working

For one thing, never use Resources.Load() as T Always use the Resources.Load<T>() form, in this case with a T of GameObject. The reason: using Resources.Load("Foo") as GameObject might load a material or something else named “Foo” that is not a GameObject. Then when you cast-as to GameObject it will come out mysteriously null and cause errors. More insidiously, if you have Foo.png and Foo.prefab, it is a 50/50 chance of what you’ll get, and it may vary from install to install. Don’t tempt fat…

And the same goes for Resources.LoadAll<T>();

Generally you want to avoid writing code that is simply waiting to mysteriously fail simply by the future appearance of an identically-named file of a different type.

ALSO: Always put your resources in type-ish subfolders if you use Resources.LoadAll<T>();. Here’s why:

As of this writing, if you use Resources.LoadAll<T>("foldername/");, Unity will load ALL FOUND ASSETS OF ANY TYPE at that path into memory, then check each type and only return to you the ones that meet type T.

But Unity will have already loaded all the assets it finds in that path, so expect massive performance problems if you don’t use subdirectories, even crashes due to running out of memory if you have a lot of assets there.

[spiney199]

I think more important is why an object constructor is causing Resources.LoadAll to be called at all. I question why save data would even need to use Resources.

I guess they’re trying to load items?

Though if it’s something like collectibles, just use a basic ID system, and check if the ID of a collectible is present in the save data to know whether its been collected or not.