strtok never gets NULL value, and ignores code after the while loop-CodePudding

I am writing a program right now that tokenizes multiple strings from a text file.

I have a while loop that continues to run until the end of the file. And nested in that loop, I have another while loop using strtok() to tokenize strings.

But, in my nested while loop with strtok(), I test with an if statement to see if I get NULL at the end of a string, but I never receive the NULL value. After the nested loop, it seems to break out of both loops, and also skips code outside of the main while loop, and ends my program with return 0;.

#include<iostream>
#include<fstream>
#include<string.h>
#include<iomanip>

using namespace std;

int main(int argc, char* argv[])
{
    //variable declarations
    ifstream inputfile;
    ofstream outputfile;
    char line[100];
    char* ptr;

    //check how many arguments in command line
    cout << "argc: " << argc << endl << endl;

    //check if there is input and output command line argument
    if (argc < 3)
    {
        cout << "missing input and output files";
    }
    cout << argv[1] << endl;
    cout << argv[2] << endl << endl;
    //cout <<"Null: " << NULL << endl;

    //opens input file from command line argument
    inputfile.open(argv[1]);

    //opens output file from command line argument
    outputfile.open(argv[2]);

    //check if input file is open
    if (!inputfile)
    {
        cout << "file not open"<< endl;
    }


    //check for input
    while (!inputfile.eof())
    {

        //inputfile = input, getline, line = output, 100 = delimiter
        //grabs the line.
        inputfile.getline(line, 100);

        //check for line
        cout << line << endl;

        //grabs first word
        ptr = strtok(line, " ");
        cout << ptr << endl;

        //loops until null
        while(ptr!=NULL)
        {
            //grabs other words until end of line
            ptr = strtok(NULL, " ");
            cout << ptr << endl;


            if (ptr==NULL)
            {
                cout << "hit NULL" << endl;
            }

        }


        cout << "afterloop" << endl;

    }

    cout << "end";

    return 0;
}

CodePudding user response：

In:

cout << ptr << endl;

//loops until null
while(ptr!=NULL)

And again in:

cout << ptr << endl;

if (ptr==NULL)

You're printing ptr before checking if it's null. It's undefined behaviour to hand operator<< a null character pointer. Rather, it assumes that the pointer points to a valid null-terminated string.

You cannot rely on any behaviour that comes out of this. The program is free to crash, pretend to work, or anywhere in between. The compiler is free to assume this doesn't happen, which can lead to it marking code as unreachable, among other optimizations, meaning you can see some unintuitive results.

While I would suggest using a better string-splitting approach, you can fix what you have by rearranging your code a bit:

//grabs first word
ptr = strtok(line, " ");

//loops until null
while(ptr!=NULL)
{
    //     We know we have a word, now we can print it.
    cout << ptr << endl;

    //grabs other words until end of line
    ptr = strtok(NULL, " ");
    //     We can remove the print from here because it will be printed on the next iteration if we have another word.

    if (ptr==NULL)
    {
        cout << "hit NULL" << endl;
    } 
}

CodePudding user response：

As @chris's answer stated, you are invoking undefined behavior by passing NULL pointers to operator<<.

And, you should NOT use inputfile.eof() in a while loop. Loop on inputfile.getline() instead and let it break the loop when it can't read another line.

You are also not validating things very well in general, and not exiting the program when pre-conditions are not satisfactory.

Try something more like this instead:

#include <iostream>
#include <fstream>
#include <iomanip>
#include <string.h>

using namespace std;

int main(int argc, char* argv[])
{
    //variable declarations
    ifstream inputfile;
    ofstream outputfile;
    char line[100];
    char* ptr;

    //check how many arguments in command line
    cout << "argc: " << argc << endl << endl;

    //check if there is input and output command line argument
    if (argc < 3)
    {
        cout << "missing input and output files";
        return 0;
    }

    cout << argv[1] << endl;
    cout << argv[2] << endl << endl;
    //cout << "Null: " << NULL << endl;

    //opens input file from command line argument
    inputfile.open(argv[1]);

    //opens output file from command line argument
    outputfile.open(argv[2]);

    //check if input file is open
    if (!inputfile.is_open())
    {
        cout << "input file not open" << endl;
        return 0;
    }

    //check if output file is open
    if (!outputfile.is_open())
    {
        cout << "output file not open" << endl;
        return 0;
    }

    //check for input
    while (inputfile.getline(line, 100))
    {
        //check for line
        cout << line << endl;

        //grabs first word
        ptr = strtok(line, " ");
        if (ptr != NULL)
        {
            do
            {
                cout << ptr << endl;

                //grabs other words until end of line
                ptr = strtok(NULL, " ");
            }
            while (ptr != NULL);

            cout << "hit NULL" << endl;
        }

        cout << "afterloop" << endl;
    }

    cout << "end";

    return 0;
}