Following these instructions I succeeded in opening a new empty window and verified that a new wayland socket had appeared:
$ ls -alt /run/user/1000
...
-rw------- 1 craig craig 120 Oct 31 13:54 .mutter-Xwaylandauth.RN8CT0
srwxrwxr-x 1 craig craig 0 Oct 31 13:54 wayland-0
-rw-rw---- 1 craig craig 0 Oct 31 13:54 wayland-0.lock
...
$ mutter --nested --wayland &
(mutter:24172): mutter-WARNING **: 13:55:28.911: WL: unable to lock lockfile /run/user/1000/wayland-0.lock, maybe another compositor is running
$ ls -alt /run/user/1000
...
-rw------- 1 craig craig 120 Oct 31 13:55 .mutter-Xwaylandauth.M06TS0
srwxrwxr-x 1 craig craig 0 Oct 31 13:55 wayland-1
-rw-rw---- 1 craig craig 0 Oct 31 13:55 wayland-1.lock
drwxr-xr-x 3 root root 60 Oct 31 13:54 ..
-rw------- 1 craig craig 120 Oct 31 13:54 .mutter-Xwaylandauth.RN8CT0
srwxrwxr-x 1 craig craig 0 Oct 31 13:54 wayland-0
-rw-rw---- 1 craig craig 0 Oct 31 13:54 wayland-0.lock
...
The new empty window looks like an empty canvas, I am pretty sure it is surface for wayland-1.
Now I try to open up a gnome-terminal in that wayland-1 canvas
WAYLAND_DISPLAY=wayland-1 gnome-terminal
WAYLAND_SOCKET=wayland-1 gnome-terminal
however in both cases it doesn't open up in the wayland-1 canvas, but instead opens up outside of it.
How to open gnome-terminal in a nested mutter(wayland) window?
EDIT: I was able to open Firefox in the nested canvas:
MOZ_DBUS_REMOTE=1 GDK_BACKEND=wayland WAYLAND_DISPLAY=wayland-1 firefox
However, replacing firefox with gnome-terminal did not work - gnome-terminal still opened outside of wayland-1.