The "undermining of the soul" most likely refers to the deconstruction of Rational Psychology of Leibnizian school, with its soul as "metaphysically simple being", which, along with deconstruction of Rational Cosmology and Theology, was a primary task in Critique of Pure Reason. According to Kant, inferring the substantiality, simplicity, and personal identity of the soul involves concluding “from the transcendental concept of the subject, which contains nothing manifold, the absolute unity of this subject itself, of which I possess no concept whatsoever”. The centerpiece of "undermining" is the critique of Mendelsohn in the Second Paralogism. Here is Kitcher's summary of it from Kant's Transcendental Psychology:
"If Rational Psychologists are permitted to argue for the simplicity and immateriality of the soul by claiming that they do not see how a material substance could realize the unity of thought, then materialists would be free to employ the same strategy to "establish" the opposite conclusion. Since the latter do not understand how an immaterial substance could realize the unity of thought, they may claim that the soul is material".
See What are the problems with the argument for the mind-body dualism from immateriality of thoughts? for reincarnations of such reasoning (which Kitcher calls the Rational Psychologist's fallacy) in the modern mind-body problem debates.
As for the transcendental ego or transcendental subject, this usually refers to the abstracted sum total of "faculties" like sensibility, understanding, productive imagination, etc., and especially the mysterious "transcendental unity of apperception", often interpreted as Kant's formalistic stand in for Cartesian "I", present in all cognitive syntheses. Except it is not something individual, personal or emotive, like the traditional soul, and is not even derived from self-introspection, like Cartesian cogito. It is one of those "conditions of the possibility of unified experience" extracted from transcendental arguments, a formal perspective. The transcendental subject later became an object of sharp criticism and revision, first by classical German idealists, who pointed out its dubious status as a kind of thing-in-itself about which Kant nonetheless talks (Hegel later identified it with sedimented historical Geist), and then by life philosophers and existentialists, for reducing human existence to an abstract rationalist phantom.
Nor are Kant's other candidates for "soul" particularly glorified. The "empirical self" of introspection is the object of study of empirical psychology, which Kant wrote will never become a science because of the obscurity of its subject. Perhaps the closest thing to the "soul" would be the "noumenal self", the seat of moral autonomy and free will. But of that nothing cogent can be said, as it belongs to the supersensible. This aspect is "undermined" at length in the Third Antinomy and Critique of Practical Reason, see Is Kant's "noumenal self" argument on freedom flawed?
Compared to all of that matter, the "stuff" of the faculty of sensibility, "that in the appearance that corresponds to sensation", the filler of the "a priori form of outer sense", space, subject to the a priori laws of Newtonian physics with its unbreakable causal chains, Kant's ideal of what science ought to be like, has a much more solid status. But Kant's philosophy is not a dualism in any conventional sense. It is barely even a realism, that hanging on the transcendent, supersensible and unknowable thing-in-itself. To draw an ontological mind/matter distinction one would have to apply categories of experience to "things" beyond any possible experience, a no-no that the antinomies warn against.
What I think James refers to is the dualism between sensibility and understanding, which neo-Kantians eliminated and which is a kind of epistemological shadow of the mind/matter distinction in Kant. He writes after the quoted passage:
"For the thinkers I call neo-Kantian, the word consciousness today does no more than signalize the fact that experience is indefeasibly dualistic in structure. It means that not subject, not object, but object-plus-subject is the minimum that can actually be. The subject-object distinction meanwhile is entirely different from that between mind and matter, from that between body and soul."
Neo-Kantians subsumed sensibility under understanding and made it changeable, evolving, after Euclidean space as an a priori form of intuition did not work out. Friedman discusses in Parting of the Ways how this revision collapsed Kant's architechtonic of pure reason, and forced a sharpened analytic/continental divide after its diverging mendings by Carnap and Heidegger.
