Shadow DOM is a DOM structure that's hidden under a tag (for example, the <video> tag has a Shadow DOM which provides the controls and timecode UI).

Virtual DOM is (typically) a pure JavaScript object-based DOM representation that gets somehow flushed to actual DOM (or to WebGL or anything else). You as the programmer use the Virtual DOM directly and try to avoid talking to the real DOM underneath. Virtual DOM has a few advantages, but the one that stands out in today's browsers is limiting DOM thrashing where you query the DOM (requiring an expensive recalc) and then modify the DOM (invalidating what was recalc'd) in a loop.