I don't know whether the answer will serve you. At least, I found it is possible to start DOSBox, execute commands, and exit it. This allows me to run automated tests on DOSBox via Travis CI. There will still be a window though, and it does not propagate an exit code to the caller.
The command would be as follows:
dosbox exit -c [your_command_here] >>log.txt" -noconsole -exit
For this to work, you have to have a file called exit in the current directory (because exit is run last, and expected to be an external command).
I use log.txt to get feedback from the Dos program.
To my knowledge, it is not possible to run DOSBox without a window. It would be nice if it returned the exit code of its command shell. This should be easy to implement, I think, but would require a source change.
I can't upvote yet but I do think this is a useful question, as DOSBox can be used for testing in a 16-bit environment. In that sense, the question would be suitable for StackOverlow, actually.