calcyman wrote: ↑May 9th, 2021, 6:09 am
Better advice would be 'if you're submitting hauls at most once every 30 minutes, then there's no need to increase your haul size any further'.
Thanks for the insight on this. I assume your worry is overall I/O still, so if I submit 4 concurrent hauls per 2 hours, that's mostly the same as 1 haul per 30 minutes as far as you're concerned? Other than the large data size than if I'd done a single large haul on a single machine, of course.
An idea, while I'm thinking of it: A service that can run somewhere and "bundle" hauls together for transmission. The service receives hauls over the network, and then queues them until you've got about 1MB (or another configurable amount) worth of data, then fires it off for validation/etc. It could even do some pre-processing on the hauls and merge them together - I'm not sure how validation works though, and if that would affect it.
My current setup is that I've got a home server with 64 mostly-idle cores, so I have plenty of room to spin up virtual machines. Since I start them relatively close together, they're all likely to submit at the same time - I want to do as much work as possible, but I don't want to "break Catagolue," as I've so often read.
(note that on my system, 1 VM doing 62 threads is actually 20k soups/second slower than 4 VMs doing 15 threads - I could probably get even more efficiency if I go to 8 VMs at 7 or 8 threads each. I'd swear I saw some graphs somewhere...)
I'm probably going to bump each one to doing 150M/haul anyway though, just to make sure I'm within that "average" of 30+ minutes per haul.
Also, pardon the minor mess of "smaller" hauls (50M, lol) I'm making. It's not once every couple of seconds, but I keep getting more and more ideas on how to make this more efficient with what I have on hand, so I'm having to kill active searches - and I don't want to just lose the work. I'm using test mode while I'm messing with things, mostly. Honest.
Edit:
Now that I've got everything set up correctly, I've found the following based on the reported averages:
62 cores * 1 virtual machine, 62 cores total: ~40k soups/second (s/s)
15 * 4, 60c: ~55k s/s (I think - didn't keep this data specifically)
7 * 8, 56c: ~75k s/s
So with my
specific setup (an Epyc 7302 server at 32c/64t, running Hyper-V) I get some pretty significant performance boots by using fewer cores/virtual machine vs. more, AND I'm not even using as many cores. I think that going too much lower would be silly, but it's interesting nonetheless.