Finding best delimiter by size of resulting array after split Kotlin-CodePudding

I am trying to obtain the best delimiter for my CSV file, I've seen answers that find the biggest size of the header row. Now instead of doing the standard method that would look something like this:

 val supportedDelimiters: Array<Char> = arrayOf(',', ';', '|', '\t')

 fun determineDelimiter(headerRow): Char {
 var headerLength = 0
 var chosenDelimiter =' '
     supportedDelimiters.forEach {
         if (headerRow.split(it).size > headerLength) {
             headerLength = headerRow.split(it).size
             chosenDelimiter = it
         }
     }
 return chosenDelimiter
}

I've been trying to do it with some in-built Kotlin collections methods like filter or maxOf, but to no avail (the code below does not work).

fun determineDelimiter(headerRow: String): Char {
    return supportedDelimiters.filter({a,b -> headerRow.split(a).size < headerRow.split(b)})
}

Is there any way I could do it without forEach?

Edit: The header row could look something like this:

val headerRow = "I;am;delimited;with;'semi,colon'"

I put the '' over an entry that could contain other potential delimiter

CodePudding user response：

You're mostly there, but this seems simpler than you think!

Here's one answer:

fun determineDelimiter(headerRow: String)
    = supportedDelimiters.maxByOrNull{ headerRow.split(it).size } ?: ' '

maxByOrNull() does all the hard work: you just tell it the number of headers that a delimiter would give, and it searches through each delimiter to find which one gives the largest number.

It returns null if the list is empty, so the method above returns a space character, like your standard method. (In this case we know that the list isn't empty, so you could replace the ?: ' ' with !! if you wanted that impossible case to give an error, or you could drop it entirely if you wanted it to give a null which would be handled elsewhere.)

As mentioned in a comment, there's no foolproof way to guess the CSV delimiter in general, and so you should be prepared for it to pick the wrong delimiter occasionally. For example, if the intended delimiter was a semicolon but several headers included commas, it could wrongly pick the comma. Without knowing any more about the data, there's no way around that.

With the code as it stands, there could be multiple delimiters which give the same number of headers; it would simply pick the first. You might want to give an error in that case, and require that there's a unique best delimiter. That would give you a little more confidence that you've picked the right one — though there's still no guarantee. (That's not so easy to code, though…)

CodePudding user response：

Just like gidds said in the comment above, I would advise against choosing the delimiter based on how many times each delimiter appears. You would get the wrong answer for a header row like this:

Type of shoe, regardless of colour, even if black;Size of shoe, regardless of shape

In the above header row, the delimiter is obviously ; but your method would erroneously pick ,.

Another problem is that a header column may itself contain a delimiter, if it is enclosed in quotes. Your method doesn't take any notice of possible quoted columns. For this reason, I would recommend that you give up trying to parse CSV files yourself, and instead use one of the many available Open Source CSV parsers.

Nevertheless, if you still want to know how to pick the delimiter based on its frequency, there are a few optimizations to readability that you can make.

First, note that Kotlin strings are iterable; therefore you don't have to use a List of Char. Use a String instead.

Secondly, all you're doing is counting the number of times a character appears in the string, so there's no need to break the string up into pieces just to do that. Instead, count the number of characters directly.

Third, instead of finding the maximum value by hand, take advantage of what the standard library already offers you.

const val supportedDelimiters = ",;|\t"

fun determineDelimiter(headerRow: String): Char =
    supportedDelimiters.maxBy { delimiter -> headerRow.count { it == delimiter } }

fun main() {
    val headerRow = "one,two,three;four,five|six|seven"
    val chosenDelimiter = determineDelimiter(headerRow)
    println(chosenDelimiter)  // prints ',' as expected
}