This is how I would answer your questions:
1) You should have no issue with vram on a 1080Ti with 11GB. I can run 80 on a 1070 with 8GB. Something else is bottlenecking you, even if it's showing up as a graphics card bottleneck. The EQ client is OOOOOOOLD, from a time when video cards had 4mb or whatever bullshit. It does not use vram. If it IS using a bunch of vram, that's some sort of graphical setting your geforce driver is auto-optimizing that you just have to find and disable. There was some....really stupid setting that used crazy amounts of resources if set incorrectly, like set the sky to simple instead of off maybe? Looking at my client now, I don't have everything turned down to minimum, but I do have some things off. There's a list of 10 check boxes, everything between pixel shaders and bloom lighting is turned off. And max FPS is set to 60 (I vaguely recall it being important that this is set to something, maybe doesn't matter much what it is set to). There's some threads out there on stuff you can fiddle with, it isn't always as simple as setting it to the lowest possible, that sometimes will push the resource useage to the vram instead of RAM, if that makes sense.
2) Best of luck, my eyes aren't good enough for fancy monitors to matter anymore.
3) I have no regrets going for more cores over speed. In theory a faster processor clears the lanes but twice the lanes definitely works fine for multiboxing. Windows 10 especially is a total hoss at making good use of all the cores for EQ. Newer games/single instances you want speed over everything else, so if are playing lots of different games maybe that's important. Your mobo looks awesome, and reading the memory compatibility page it looks like it supports some 128GB options (be careful with batches tho, hard/expensive to find 128GB sets so it's a squeeze your nuts and pray ordered sets work together. I sorted through about ten thousand packages of 2x16 sticks to find four in a row...You might find a few more sticks of the 4x16 (hopefully not 8x8) set you already have, stick 'em in and maybe they'll work.
Your processor is also pretty great on paper. I...seriously doubt it's the bottleneck in your current build. If you can afford it though it does look like there's some drop in options with more cores, but comparing our processors, yours is just a little bit slower than mine, you ought to be able to hit 40 toons. You might could benefit from more RAM, but I think if you just configure eqwire or mq2fps you should be fine. It shouldn't just be possible it should feel smooth.
Specifics on my experience below:
I box 54 on one PC. I cannot do this without eqwire plugin. I have eqwire set to render one frame every ten minutes or so for non-active instances. With practice and fluency you really don't need to "see" out of any other windows and at 54 toons your monitors would have to be pretty large to make out much detail anyway, I do have a second monitor that has all the windows flickering, but I use it more for a "yup, every toon made it across that zoneline" than any other reason. See other's post on maybe running two machines, especially if you're savvy enough to set up remote start/launch etc from the primary machine. I know someone who ran a couple of six box crews through older laptops monitored by his BCS running on an always on raspberry pi. For a just a couple hundred bucks he was able to add basically eighteen boxes, but everything was set up to autopatch/launch/updates to client/pc/3rdparty programs. I don't have the skill for that.
My system info:
System Model X299 UD4 Pro
Processor Intel(R) Core(TM) i9-7920X CPU @ 2.90GHz, 2904 Mhz, 12 Core(s), 24 Logical Processor(s)
BaseBoard Manufacturer Gigabyte Technology Co., Ltd.
Platform Role Desktop
Installed Physical Memory (RAM) 128 GB
Adapter Description NVIDIA GeForce GTX 1070
The graphics card doesn't max out until I load closer to eighty instances.
On my old build, 64GB of RAM would only get me to around 42 characters, and 36 was the max for smooth running, which is a major reason I switched to the different socket to open up a larger RAM reservoir. I find EQ works best when it has around 700mb of RAM per instance, but you also need some cushion for OS and other things. You *can* force down some of the RAM use, I've made it work at 250mb per instance, but it will still end up being a bottleneck for you trying to go to 40 regardless of other system parameters, in my experience. And it felt like the more I tried to stop the client from using RAM, the more stress it put on the cpu/vcard.
The 7920X was really cool because it upped everything to 44 threads (plus it was twice the cores of my previous setup), with 54 characters and netflix running I generally have 30GB of RAM leftover. Without eqwire I max out at 36 characters before performance slows enough to notice.
But the biggest quality of life improvement, I'm saying, the secret to shout from the rooftops: Get an NVME M.2 SSD. Sweet merciful christ. I can't necessarily run more toons, but I bet it saves me an hour a day. 54 characters zone, not instantly, but it FEELS instantly compared to how it used to be. I used to just go take a break, stretch, use the bathroom, get a snack, every time I zoned. Now? I'm pretty sure it's server-lag is the only reason it isn't instant. You've already got an NVME so you're all set there.
My next build (parts are on the way) will upgrade to a 10980XE for more cores and threads, bigger cache, and two of the newest NVME's which are listed as twice as fast, I'm going to try them in a RAID configuration and see if I can get 4x the old speed.
I'm keeping the old board, which limits me to 128GB of RAM still, as I don't think the RAM is bottlenecking me anymore and 256 GB of RAM won't let me stop using eqwire, maybe in 2025...
And I'm not going for faster RAM because I haven't noticed that making a difference either (EQ uses just as much DDR4 3000 RAM per instance as it used DDR3 1200 RAM or whatever my old sticks were seems like).
If I can ever FIND an RTX card in stock anywhere I'll see how they do. I wasn't thinking EQ made enough use of the graphics card back when I was seeing the 20 series cards, and everything is completely sold out now that the 30 series launched. The 1070 is still cranking away, I'm very satisfied with that purchase though I was nervous at the time it might not be enough. Couldn't find any 1080Ti's in stock at the time so just went for it. Again, I can't imagine your 1080Ti is really maxxing out, maybe if you were playing with all settings turned to max on each account without eqwire or mq2fps, even then, I'd expect something else was getting full up and causing the graphics card to have to work too hard.
I know this is late but just thought I'd share my journey. I don't even really like playing EQ that much anymore, but I DO get a lot of satisfaction getting a single PC up to running a whole raid. Something about a childhood spent barely getting a single instance of the game to run at all. It's also exceedingly satisfying pointing 40 melee characters at something and watching it explode.
