Skip to content

Improve SQLite write speed, indefinite hang while analyzing large bundles (>1 GB) #66

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
UnityZappy wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Improve SQLite write speed, indefinite hang while analyzing large bundles (>1 GB) #66

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 20 additions & 63 deletions Analyzer/PPtrAndCrcProcessor.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
using Force.Crc32;
using UnityDataTools.FileSystem;

namespace UnityDataTools.Analyzer;

// This class is used to extract all the PPtrs in a serialized object. It executes a callback whenever a PPtr is found.
// It provides a string representing the property path of the property (e.g. "m_MyObject.m_MyArray[2].m_PPtrProperty").
/// <summary>
/// Walks serialized object TypeTrees to extract PPtr references and compute a rolling CRC32.
/// External stream segments (StreamingInfo / StreamedResource) extend the CRC using offset, size, and path only,
/// avoiding full reads of large companion .resS data.
/// </summary>
public class PPtrAndCrcProcessor : IDisposable
{
public delegate int CallbackDelegate(long objectId, int fileId, long pathId, string propertyPath, string propertyType);
Expand All @@ -18,65 +19,31 @@ public class PPtrAndCrcProcessor : IDisposable
private long m_Offset;
private long m_ObjectId;
private uint m_Crc32;
private string m_Folder;
private StringBuilder m_StringBuilder = new();
private byte[] m_pptrBytes = new byte[4];

private CallbackDelegate m_Callback;

private Dictionary<string, UnityFileReader> m_resourceReaders = new();
Copy link
Copy Markdown
Collaborator

@SkowronskiAndrew SkowronskiAndrew Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if this existing implementation was very expensive in terms of allocating a lot of memory during the processing and never freeing it. Is it possible you were running out memory?
Apparently each UnityFileRead allocates 4MB and i guess it might grow based on the actual data size?

I'm guessing that the way to fix things will be to improve the management of these UnifyFileReaders.
I think these should be flushed between SerializedFiles - currently .resS and .resource files are NEVER shared, its
always pointing at another companion file with the same filename root.

So the only purpose of caching would be while processing different objects of the same serialized file (because they might reference different segments of the same file)

I recommend that you to take a look at the access pattern, e.g. using some logging to double check exactly what requests are coming in for the external streams in the case that was so slow.


public PPtrAndCrcProcessor(SerializedFile serializedFile, UnityFileReader reader, string folder,
CallbackDelegate callback)
public PPtrAndCrcProcessor(SerializedFile serializedFile, UnityFileReader reader, CallbackDelegate callback)
{
m_SerializedFile = serializedFile;
m_Reader = reader;
m_Folder = folder;
m_Callback = callback;
}

public void Dispose()
{
foreach (var r in m_resourceReaders.Values)
{
r?.Dispose();
}

m_resourceReaders.Clear();
}

private UnityFileReader GetResourceReader(string filename)
/// <summary>
/// Extends CRC32 with a stable fingerprint for an external stream segment without reading blob bytes.
Copy link
Copy Markdown
Collaborator

@SkowronskiAndrew SkowronskiAndrew Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CRC is stored in the database to represent the content of the object, including all external resource references like mesh and texture content. The CRC is the fingerprint for "has this asset changed?"

If a texture blob in a .resS file changes but stays the same size, the old code detects it (different bytes =
different CRC)

The new code gives an identical "fingerprint" (same offset + size + filename)

It sounds like there are performance problems in the existing calculation but i don't think this is the solution.

Probably there are two things to do:

  1. Continue to calculate the CRC of .resS and .resource blobs, but make sure we do this in an efficient fashion. I'll take a bit more look at the existing code.

  2. Offer a way to skip the CRC calculation if you don't need it. Currently you can turn it off with --skip-references, but that is not intuitive, so there should be a separate --skip-crc and some careful work to make sure that the permutations of those flags work properly.

/// </summary>
private static uint AppendExternalStreamFingerprint(uint crc32, long offset, int size, string filename)
{
var slashPos = filename.LastIndexOf('/');
if (slashPos > 0)
{
filename = filename.Remove(0, slashPos + 1);
}

if (!m_resourceReaders.TryGetValue(filename, out var reader))
{
try
{
reader = new UnityFileReader("archive:/" + filename, 4 * 1024 * 1024);
}
catch (Exception)
{
try
{
reader = new UnityFileReader(Path.Join(m_Folder, filename), 4 * 1024 * 1024);
}
catch (Exception)
{
Console.Error.WriteLine();
Console.Error.WriteLine($"Error opening resource file {filename}");
reader = null;
}
}

m_resourceReaders[filename] = reader;
}

return reader;
crc32 = Crc32Algorithm.Append(crc32, BitConverter.GetBytes(offset));
crc32 = Crc32Algorithm.Append(crc32, BitConverter.GetBytes(size));
crc32 = Crc32Algorithm.Append(crc32, Encoding.UTF8.GetBytes(filename));
Comment on lines +41 to +45
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AppendExternalStreamFingerprint uses BitConverter.GetBytes(...), which is endianness-dependent. That makes the computed CRC vary on big-endian platforms even for identical data. Prefer writing offset/size in a fixed byte order (e.g., little-endian via BinaryPrimitives) before appending to the CRC to keep results stable across architectures.

Copilot uses AI. Check for mistakes.
return crc32;
}

public uint Process(long objectId, long offset, TypeTreeNode node)
Expand Down Expand Up @@ -123,7 +90,7 @@ private void ProcessNode(TypeTreeNode node, bool isInManagedReferenceRegistry)
if (node.Children.Count != 3)
throw new Exception("Invalid StreamingInfo");

var offset = node.Children[0].Size == 4 ? m_Reader.ReadInt32(m_Offset) : m_Reader.ReadInt64(m_Offset);
var streamOffset = node.Children[0].Size == 4 ? m_Reader.ReadInt32(m_Offset) : m_Reader.ReadInt64(m_Offset);
m_Offset += node.Children[0].Size;

var size = m_Reader.ReadInt32(m_Offset);
Expand All @@ -136,12 +103,7 @@ private void ProcessNode(TypeTreeNode node, bool isInManagedReferenceRegistry)

if (size > 0)
{
var resourceFile = GetResourceReader(filename);

if (resourceFile != null)
{
m_Crc32 = resourceFile.ComputeCRC(offset, size, m_Crc32);
}
m_Crc32 = AppendExternalStreamFingerprint(m_Crc32, streamOffset, size, filename);
}
}
else if (node.Type == "StreamedResource")
Expand All @@ -162,12 +124,7 @@ private void ProcessNode(TypeTreeNode node, bool isInManagedReferenceRegistry)

if (size > 0)
{
var resourceFile = GetResourceReader(filename);

if (resourceFile != null)
{
m_Crc32 = resourceFile.ComputeCRC(offset, size, m_Crc32);
}
m_Crc32 = AppendExternalStreamFingerprint(m_Crc32, offset, size, filename);
}
}
else if (node.CSharpType == typeof(string))
Expand Down Expand Up @@ -301,19 +258,19 @@ bool ProcessManagedReferenceData(TypeTreeNode refTypeNode, TypeTreeNode referenc
throw new Exception("Invalid ReferencedManagedType");

var stringSize = m_Reader.ReadInt32(m_Offset);
m_Crc32 = m_Reader.ComputeCRC(m_Offset, (int)(m_Offset + stringSize + 4), m_Crc32);
m_Crc32 = m_Reader.ComputeCRC(m_Offset, stringSize + 4, m_Crc32);
var className = m_Reader.ReadString(m_Offset + 4, stringSize);
m_Offset += stringSize + 4;
m_Offset = (m_Offset + 3) & ~(3);

stringSize = m_Reader.ReadInt32(m_Offset);
m_Crc32 = m_Reader.ComputeCRC(m_Offset, (int)(m_Offset + stringSize + 4), m_Crc32);
m_Crc32 = m_Reader.ComputeCRC(m_Offset, stringSize + 4, m_Crc32);
var namespaceName = m_Reader.ReadString(m_Offset + 4, stringSize);
m_Offset += stringSize + 4;
m_Offset = (m_Offset + 3) & ~(3);

stringSize = m_Reader.ReadInt32(m_Offset);
m_Crc32 = m_Reader.ComputeCRC(m_Offset, (int)(m_Offset + stringSize + 4), m_Crc32);
m_Crc32 = m_Reader.ComputeCRC(m_Offset, stringSize + 4, m_Crc32);
var assemblyName = m_Reader.ReadString(m_Offset + 4, stringSize);
m_Offset += stringSize + 4;
m_Offset = (m_Offset + 3) & ~(3);
Expand Down
5 changes: 4 additions & 1 deletion Analyzer/SQLite/Writers/SQLiteWriter.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,16 @@ public void Begin()
try
{
m_Database.Open();

using var walCommand = m_Database.CreateCommand();
walCommand.CommandText = "PRAGMA journal_mode=WAL";
walCommand.ExecuteNonQuery();
Comment on lines +38 to +40
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PRAGMA journal_mode=WAL set here is later overridden by Resources.Init (Init.sql currently ends with PRAGMA journal_mode = MEMORY;), so the connection will not actually run in WAL mode. To make WAL effective, either move the WAL pragma to after executing Resources.Init, or remove/adjust the journal_mode pragma in Init.sql to not override the desired mode.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Collaborator

@SkowronskiAndrew SkowronskiAndrew Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems this is a good optimization to add for UnityDataTools, but according to the copilot analysis this doesn't actually apply it.

}
catch (Exception e)
{
Console.Error.WriteLine($"Error creating database: {e.Message}");
}

// this does all the legacy import of Init.sql
using var command = m_Database.CreateCommand();
command.CommandText = Resources.Init;
command.ExecuteNonQuery();
Expand Down
2 changes: 1 addition & 1 deletion Analyzer/SQLite/Writers/SerializedFileSQLiteWriter.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ public void WriteSerializedFile(string relativePath, string fullPath, string con
{
using var sf = UnityFileSystem.OpenSerializedFile(fullPath);
using var reader = new UnityFileReader(fullPath, 64 * 1024 * 1024);
using var pptrReader = new PPtrAndCrcProcessor(sf, reader, containingFolder, AddReference);
using var pptrReader = new PPtrAndCrcProcessor(sf, reader, AddReference);
int serializedFileId = m_SerializedFileIdProvider.GetId(Path.GetFileName(fullPath).ToLower());
int sceneId = -1;

Expand Down
17 changes: 10 additions & 7 deletions UnityFileSystem/UnityFileReader.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,16 +117,19 @@ public byte ReadUInt8(long fileOffset)
return m_Buffer[offset];
}

/// <summary>
/// Computes CRC32 over a contiguous byte range, reading the file in buffer-sized chunks.
/// </summary>
public uint ComputeCRC(long fileOffset, int size, uint crc32 = 0)
{
var readSize = size > m_Buffer.Length ? m_Buffer.Length : size;
var readBytes = 0;

while (readBytes < size)
var remaining = size;
while (remaining > 0)
{
var offset = GetBufferOffset(fileOffset, readSize);
crc32 = Crc32Algorithm.Append(crc32, m_Buffer, offset, readSize);
readBytes += readSize;
var chunk = (int)Math.Min((long)m_Buffer.Length, remaining);
var offset = GetBufferOffset(fileOffset, chunk);
crc32 = Crc32Algorithm.Append(crc32, m_Buffer, offset, chunk);
fileOffset += chunk;
remaining -= chunk;
}
Comment on lines +120 to 133
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ComputeCRC chunking logic was changed to fix offset/size handling; this is easy to regress without coverage. Add a unit test that verifies CRC results for ranges that cross the internal buffer boundary (e.g., size > buffer, and a final partial chunk) to lock in the corrected behavior.

Copilot uses AI. Check for mistakes.

return crc32;
Expand Down
Loading