C # using a List <string> quote OutOfMemoryException consult!-CodePudding

This is the external TXT file, a total of 500 m. for data to the operation, so I want to read all these data to List , but the program read 500000 rows thrown OutOfMemoryException abnormalities, ask how to deal with, code idea is very simple, is to read each line, and then use the Split read each data, in the for loop cutting out abnormalities,
List ListZcorn=new List (a);
Using (StreamReader sr=new StreamReader (" XXX. TXT ", System. Text. Encoding. The Default))
{
String readline="";
While (readline. The Trim ()!="")
{
Readline=sr. Readline ();
ListZcorn. Add (readline);
}
}
List Listarray=new List (a);
For (int I=0; I & lt; ListZcorn. Count; I++)
{
Ch=listZcorn string [] [I] the Split (new string [] {" "}, StringSplitOptions. RemoveEmptyEntries);
Listarray. AddRange (ch);
}

CodePudding user response:

Tell you the memory ~ so large amount of data batch it all into memory, who stand, processed part to empty the list and then deal with the following

CodePudding user response:

In the while loop can be done for operation, do not need to once again the list, or remove handled in the for character set a try,

CodePudding user response:

Your memory is stored so much character, can ask to change the computer

CodePudding user response:

Change idea, must change, use database pagination, read 1000, processed, read the rest

CodePudding user response:

If you hard needs to be read, can only add memory no other way.

CodePudding user response:

Out of memory, is divided into several batch string

CodePudding user response:

You read this, 500 m, and then break up, produce a large amount of debris...

CodePudding user response:

Unless the 500 m is a large XML, or a json, you have to read it out to whole treatment, that can only be so

Otherwise, it is to read while processing edge to release,

Maybe I said is hard to understand, that me a everyone so processing, Tcp

Suppose you this 500 m is a TCP data sent to me half an hour, according to what you have said something interval for each command (you said you splite), so I should wait for half an hour, and then the whole splite, obviously not, any written of TCP didn't do this.

CodePudding user response:

Question the reason is clear, there are a lot of people say: out of memory,
By default the.net application memory usage limitation, can only use 2 gb of memory, no matter you are how much physical memory,
In this case, there are two kinds of processing methods:
1, batch, all the above said, also have nothing to say, the partial storage or partial specifically to read, this is the top priority, consider to have the algorithm meets the requirements;
2, must want to deal with large files, can only do it: modify the application configuration file:

Must generate a 64 - bit application, it is necessary to run on 64 - bit operating system, lack of physical memory is memory,

CodePudding user response:

StringBuilder you put in will be better, if you want to load in,

CodePudding user response:

If is $500 m at the same time to participate in the operation (such as sorting), in addition to increasing physical memory, there are ways to give it a try, in a SQL database, the arithmetic for SQL server