Capacity Limit on SyntaxEdit control

Is there a limit to the amount of text that can be loaded into a SyntaxEdit control? We have a customer that is attempting to load a 2gb+ file into a SyntaxEditor and our application crashes with an System.OutOfMemory exception when we use the SyntaxEdit.LoadFile() method to load the text as well as when we have a StringBuilder object with the 2gb of test in it and assign the SyntaxEdit’s Text property to the StringBuilder.toString().

Hi Scott,

The short answer is that setting 2 GB text buffer to the editor’s text will most likely use all physical memory.
Storing the buffer itself and handling such amount of the text in the editor’s TextSource will need few times more of the memory to be allocated (and if you have Prefer32Bit flag for application on, the memory that application can access is limited to 3GB)

You can set edit.Source.OptimizedForMemory to true to reduce it slightly, but it will still take a lot. You can definitely handle OutOfMemory exception when setting editor’s text, but in case this exception occurs, there is not much you can do except of clearing editor’s text.

We’ve done some experiments to test these limits and here are our findings:

  1. Tested with AnyCPU/Prefer32Bit, file size about 500MB

This code generates strings in memory:

      IList<string> lines = new List<string>();

        public Form1()
        {
            for (int i = 0; i < 25000000; i++) 
            {
                lines.Add("sample text " + i.ToString());
            }

and this code writes them to the file:


            if (!File.Exists(path))
            {

                var file = new StreamWriter(path);
                foreach (string line in lines)
                {
                     file.WriteLine(line);
                }
                file.Close();
            }

Physical memory allocated for for 500MB file is about 1.5GB. Since .NET strings are Unicode-based, there are 2 bytes allocated for every character. So 1GB would be a minimum. Every string has a header that adds up another 18 bytes and an array of integers would take another 4 bytes per string - that explains extra 500MB.

Change this number to 500.000.000 and you will have 1GB file when written and 2.9 GB in memory.

If you change this number to 800.000.000, you will get a out of memory error, since 32-bit applications can not allocate more than 3 GB anyway (even with IMAGE_FILE_LARGE_ADDRESS_AWARE).

Now if you switch Prefer32Bit off, this would allow to use any physical memory available under x64 bit Windows, but it will increase size of IntPtr to 8.

So 500MB file will take about 1.9 GB in memory, 1GB file will take about 4.4 GB in memory and 2GB - more than 8GB.

These numbers are based on above example, real log files most likely will have longer lines, therefore there will be less amount of lines in total (and slightly less additional memory taken by string headers)

  1. Now, similar test for TextSource/SyntaxEdit:

We do not have a virtual-mode for loading only portion of file into the editor, so internally it’s the same list of lines. On top of that, for every character in string we allocate the array of 2-bytes character for color-related information (which in case of log files is not really required), plus some additional information per every line.

    public class StringItem : IStringItem
    {
        private string str;
        private int lexState;
        private int prevLexState;
        private ItemState state;
        private StringItemInfo[] textData;
...
}

So memory taken by TextSource/SyntaxEdit when loading the same file would be more than twice as much as just for list of strings.

You can decrease by setting edit.Source.OptimizedForMemory property to true, in this case color information is created on-demand and typically uses less memory (especially if lexer is not set)

   public class LowFootprintStringItem : IStringItem
    {
        private string str;
        private int lexState;
        private int prevLexState;
        private ItemState state;
        private StringFragment[] fragments;
...
}

Therefore with Prefer32Bit enabled it could only load up to 400MB file and will allocate about 3GB.

With OptimizedForMemory for memory 500MB file will take about 2.7GB when loaded in the editor.

With Prefer32Bit off and OptimizedForMemory on, 1GB file will take about 8GB of memory when loaded in the editor. If you’re setting text using string buffer, this amount of memory will add up to the text buffer size.

Realistically, the only thing you can do here is to set OptimizedForMemory, and then use temporary files instead of large text buffers to set editor’s text. And Prefer32Bit needs to be switched off, otherwise you will hit memory limits pretty quickly.

Regards,
Dmitry

Thank you Dmitry for the comprehensive response to my question. There is a lot to digest there but I wanted to express my appreciation for the obvious amount of work you put into the response.

I am trying to figure out how to catch the OutOfMemory exception. I changed the project to no longer prefer 32 bit and to optimize the SyntaxEdit control for memory. I also changed the approach to having the SyntaxEdit object load the log file directly using LoadFile().

             try
            {
                syntaxEdit.LoadFile(fileName);
            }
            catch (OutOfMemoryException e)
            {
                MessageBox.Show("There is insufficient memory to load this file", "Insufficient Memory");
                syntaxEdit.Text = "";
            }
            catch (Exception e)
            {
                MessageBox.Show(e.Message, "Exception Encountered");
            }

When I try to load the huge log file, the OutOfMemory exception is not caught and our application crashes. What am I doing wrong here?

Thanks for any help you can provide.

Hi,

As per this MSDN article: OutOfMemoryException Class (System) | Microsoft Docs,
in general, application execution cannot be continued after OutOfMemoryException had occured. There are some exceptions like in case of StringBuilder construction, but the general advice is similar to this (quoting the article linked above):

      catch (OutOfMemoryException e)
      {
         Console.WriteLine("Terminating application unexpectedly...");
         Environment.FailFast(String.Format("Out of Memory: {0}",
                                            e.Message));
      }

I.e. the catch clause is used here to log the event and FailFast the process explicitly.

Yevgeni,

Thank you for your reply.

My question is more about why my code implementation does not actually catch the OutOfMemory exception. Our goal here is to catch the exception and shut down the application cleanly (the FailFast will undoubtedly be part of that process) rather than have our application crash and look unprofessional.

For a case when the exception appears somewhere in the message loop processing code, you could try handling an unhandled Windows Forms exception as it is shown in the example in the following article:
Application.SetUnhandledExceptionMode Method (System.Windows.Forms) | Microsoft Docs