Swashplate engines are essentially the current go-to: they're extremely compact and have many other good properties, but wear out quickly. I believe they are significantly more compact than rotaries (and this), though.