Current situation - On pressing 1,2,3 on the keyboard, it toggles through 3 different scenes.

The problem:

1. For instance, during the runtime, I press: 1 -> (Scene 1), 2 -> (Scene 2), 3 -> (Scene 3) and then again: 1 -> (Scene 1), its loading the scene but also makes a copy of that scene.

A screenshot of the Hierarchy

My Scene Loading code:

    private IEnumerator SceneLoader(string sceneName)
{
    if (!SceneManager.GetSceneByName(sceneName).isLoaded)
    {
        currentScene = SceneManager.GetActiveScene();
        AsyncOperation loadScene = SceneManager.LoadSceneAsync(sceneName, LoadSceneMode.Additive);

        asyncScene = loadScene;

        while (!asyncScene.isDone)
        {
            Debug.Log("Scene Loading: "   loadScene.progress);
            yield return null;
        }

        Scene getScene = SceneManager.GetSceneByName(sceneName);

        while (!getScene.isLoaded)
        {
            Debug.Log("Scene loading!!");
            yield return null;
        }

        Debug.Log("Scene name: "   getScene.name);
        if (getScene.IsValid())
        {
            SceneManager.MoveGameObjectToScene(this.gameObject, getScene);
            SceneManager.SetActiveScene(getScene);
            ShiftItems(getScene);
        }


        Debug.Log("Scene name to unload: "   currentScene.name);
        SceneManager.UnloadSceneAsync(currentScene);
    }
    yield return null;
}

Update: After debugging more into this issue, I realized this: For instance, you have 3 scenes: Main Menu (Scene 1), Settings (Scene 2) and Pause Menu (Scene 3). Now what I was doing is that I was having all the Game Objects (which were to be moved to the other scenes) in the Main Menu. Now when I toggled through the scenes and go from Scene 3 to Scene 1 again... It seemed like it was taking inputs from 2 duplicated objects which had the input script attached and the starting scene was never unloaded for some reason I dunno yet. There is something fishy about the start point.

Fix: What I did?

1. Create an empty scene.
2. Get all the Game Objects you want to keep throughout the different scenes in there.
3. Attach a script to a game object which in my case its the scene manager where you change the scene: Empty Scene -> Scene 1 (In my case its the main menu scene)

So, since the issue was something related to the starting scene, with this approach you won't have to return to the starting scene since that's the empty scene

Hope this helps