Mrthfx

First off, you don't need to point out that the current generation home game consoles use APUs with x86_64 cores. Re-read the question title, if necessary.

I cannot recall a single arcade system or game console that ever used x86 for its CPU. I'm happy to be corrected in the comments if there were some. Notwithstanding these exceptions, which must be incredibly rare, it sure seems that gaming hardware steered away from this otherwise incredibly popular CPU family.

Why is this the case? What tradeoffs in game hardware design likely led engineers away from using x86 for the CPU?

The original 8086 was quickly overshadowed by the Z80, which was somewhat compatible but easier to work with as it required less support hardware. Also many arcade developers preferred the 6502 and derivatives, and then later the 68000 which was easier to work with on both the hardware and software fronts.

Another issue was that the development machines available for testing code were often 68000 based as well. One prime example was the Sharp X68000. A lot of game programmers of that era were self taught too, and for hobbyist home computer systems Z80 and 6502 dominated with very few using 8086.

Finally, the 8086 was much more expensive than the Z80, while offering no real advantages over it unless you were expecting to buy millions and sell a line of compatible computers for years to come.

Video game hardware, whether for home consoles or arcade machines, is designed pretty much from scratch. Hardware designers have pretty much free reign on choosing what CPU to use, basing their choice factors like cost and ease of programming. The Intel 8086, quite frankly, was a poorly designed processor and was never well regarded. While you could argue it made reasonable compromises at the time it was released (1978), these compromises ended up hanging around its neck like an albatross. If IBM hadn't picked the Intel 8088 for its Personal Computer in 1981, you probably wouldn't be asking this question.

We take the x86 architecture for granted today, but before the IBM PC it was fairly obscure and afterwords widely ridiculed. In particular, it compared poorly with the Motorola 68000 which had a flat 24-bit address space, more orthogonal instruction set and 16 32-bit registers. The 8086 used a segmented 20-bit address space, placed more restrictions on how various registers could be used, and only had 8 16-bit registers. It also wasn't particularly cheap, though the 8088 with its 8-bit data bus helped reduce overall costs compared to the 8086.

During the 70's and first half of the 80's, 16-bit CPUs like the 8086 and 68000 weren't really much of a consideration. The games of this era didn't demand anything more powerful than an 8-bit Z80 or 6502. While there were Gottlieb/Mylstar arcade games like Q*Bert in the early 80's that used a 5 Mhz 8088 CPU, it's not clear what advantage this gave the machines. Performance in games of this era was mostly limited by how fast the CPU could access memory. Because of how the 8086/8 was designed, this made the 8088 effectively about as fast as a Z80 or 6052. These Gottlieb/Mystar arcade games also only had 64k (16-bit) memory maps, so didn't benefit the 8088's 20-bit address space.

Starting around the mid-80's, games had started moving beyond the capabilities of 8-bit CPUs. While the dominance of the IBM PC in the personal computer market at this point would've meant there would be programmers out there familiar with the 8086, there would've been few people singing its praises. By and large 68000 CPUs were chosen for new arcade game hardware designs that needed more power than 8-bit CPUs offered. Console hardware being more cost sensitive stuck with 8-bit CPUs for the rest of the decade, though most of the next generation went with 16-bit CPUs, either the 68000 or 65816. It's also worth mentioning that the two major new home computer designs of the mid-80's, the Commodore Amiga and Atari ST, also went with the 68000.

While arguably the 80386, introduced in 1985, solved a lot of the 8086's problems, with a more orthogonal instruction set, 32-bit flat address space and 32-bit registers, it wasn't until the early 90's that games started demanding the level of performance it offered, and when its price would dropped to make it competitive in new hardware designs. It's not entirely clear to me why it didn't attract more interest at this point. The early 90's was also about the time that the IBM PC became the premier platform for home computer gaming. It would've inherited the disdain its predecessors had, but there were some arcade boards designed in the early 90's that use the 8086-compatible NEC V30 type CPUs. I think the main factor against at the time was back then RISC-based architectures were considered the future, while CISC-based architectures like the x86 and 68k considered obsolete. Still that didn't stop Sega from using the CISC-based NEC V60 CPU its arcade hardware designs in the early 90's.

For the rest of the 90's though, RISC-based CPUs like the Hitachi SH and IBM PowerPC, dominated arcade hardware designs, at least at the high performance end. At the lower performance end, cheaper 68k and NEC V30 based designs were still in use. In the home console market the 5th generation was almost all RISC CPUs, though notably the Japan-only FM Towns Marty used an AMD 386SX CPU. For the most part, this situation continues to around the turn of the century, with both arcade games and the 6th generation of consoles.

A big exception is Microsoft's Xbox. A 6th generation console, released in 2001, it has an Intel Pentium III CPU, much like PCs of the time. It's not surprising that Microsoft, with its long experience using the x86 CPU made this design choice, but its only a few years after this that mainstream Intel and AMD CPUs start appearing in arcade hardware. Although these x86-based arcade machines aren't really new hardware designs, they're PC clones running Windows or Linux. The 7th generation of home consoles went exclusively with PowerPC CPUs, but I suspect this had more do with the prices IBM was offering rather than the relative technical merits of the CPUs. Arcade games went increasingly with PC clone based hardware.

Today the choice of CPU in current game hardware designs is unremarkable. Home consoles and arcade games use x86 CPUs just like our personal computers do. Handheld consoles use ARM CPUs just like our phones do.

So, in the early days of game hardware design, x86 CPUs weren't chosen simply because there wasn't good reason to use one except for IBM PC compatibility. Later 32-bit x86 CPUs solved a lot the architecture's problems but RISC CPUs were seen as more modern. Today the ubiquity x86 architecture combined with its unrivalled speed has turned it into the dominant CPU architecture for game hardware that doesn't need to run off a battery.

Konix Multisystem: 6 MHz 8086 (1989).

Sure, it was cancelled just before release, but it got amazing press (I remember Jeff Minter raving about it at Earls Court) and some of it lived on as the (68k based) Atari Jaguar.