It does exactly as described. Sometimes it will shift to the second window as intended. Often it will skip to the third window open instead, requiring one to continue cycling back to the second window.

You can Alt+Shift+Tab to go in reverse direction.

Or just release Tab while keeping Alt depressed and navigate the thumbnails with the arrow keys.

I think it only happens if you use Edge

Oh okay, I use FF. This must be why. I alt+tabbed 100+times and didn't experience the issue. Didn't edge tabs now recently show in alt+tab? I wonder if it's because they are surfing between two tabs and alt tab is showing the last tab they clicked on rather than the last window they had up.

FF uses CTRL+Tab to cycle through tabs like alt tab.