Review my thread pattern?

Performance Tuning
#1

I’m currently building an implementation of the Notchian Minecraft server and I wanted to run my thread pattern past you guys :stuck_out_tongue:


1. Entity updating thread pool, # of threads is number of processors; this loops through all connected entities and updates them...updates are placed in a queue every tick and emptied at the end, runs at 50ms
2. File I/O single thread, saves all world(s) and players to file, can be queued by the server GUI to execute an unexpected save, runs at 3 minutes
3. Server socket mgmt. single thread, selects all currently selectable channels and sends them to appropriate method: accept, read, or write, runs at 600ms
4. Server packet flow mgmt. single thread, grabs all upstream packet requests in its queue and processes them, followed by sending all downstream packets in the queue to appropriate clients, it does this by polling every Session object's instance for its queue, runs at 50ms
5. World update single thread, contributes to packet flow queue by consuming Task objects that can be converted into downstream packets, updates world time, runs at 50ms

Unfortunately, most threads must run at 50ms because that is how Minecraft is built to work (time has to be updated every 50ms, and there’s no use in dedicating one thread to updating a long for the sake of not having two other threads run at the same tick). I would much prefer something like 10/20ms, because my machine is quite incredible and is comfortable with pretty much anything :slight_smile:

I’m asking you guys to tell me what you think of this thread pattern for a game like Minecraft…will it encounter a problem, will it perform slowly, etc.

Thanks ;D

#2

how does it make sense to multi thread the player updating but single thread your IO (reading/writing of packets, adding to queue, that part)

unless I misinterpreted

#3

I haven’t had time to think about file I/O ops yet…now that I think about it, I don’t think it would be that easy to execute a lot of I/O operations on one thread.

Looking at how I’ve mixed my pattern up a bit (I run using Netty networking framework now, so its threads are accounted for) I’m now looking at something like this (not including Netty) :

Entity updating ~ Number of threads equals number returned from Runtime#availableProcessors()
File I/O ~ Number of threads equals number returned from Runtime#availableProcessors() OR I might do that number / 2, depends on which works better
World updating ~ One thread per running world, this works fine because Minecraft servers often have multiple worlds running, and the amount of tasks executed here is small

On my beta MC server, I will be running one world for a while until I feel it necessary to test more than one, which means I’ll be running 9 non-networking threads on my 2-core hyperthreading processor @ 2.93 GHz. If it can run a CraftBukkit server with 25 people just fine (~20% CPU) it can run mine :wink:

ALSO

Not relevant to my current situation, but…

You said it would be bad for networking packet I/O, but I’ve seen (and talked to the people who’ve made) gaming servers that run Networking I/O operations on one thread each. As long as that thread is dedicated to a specific operation, and you do all you can to make it do as little as possible while still doing its job, you can easily get it done.

#4

The disk can only do one thing at a time, multithrading IO doesn’t make sense. At least as far as I know.

#5

It’s even worse, if you’re doing multi-threaded (heavy) file I/O on a HDD (mechanical seeks) you’ll ruin your performance, you can easily lose factor 25x or so, when one file is near the center and the other is near the edge of a disk.

#6

That doesn’t sound right. If you multi-thread your file requests, you could improve performance many times on both mechanical HDs and SSDs. Sure, if you’re just reading large files, reading 32 of them at the same time is gonna reduce performance on mechanical drive, but reading 32 4kb files at the same time may increase performance a lot due to how a mechanical hard drive reads files in the order they are encountered when seeking, not in the order they were requested. It all depends on the file size of the files you’re reading. If you have many small files, multiple threads are going to have better performance.

Even if you are reading larger files, using 2 or more threads might be a good idea, as that might remove the HD - CPU sync that happens when reading:
CPU requests file -> CPU idles -> HD seeks and finds file -> HD sends file data to CPU -> HD idles -> CPU starts processing file…
It’s the same reason you queue up OpenGL commands and not wait for a command to complete, right? I might be off on this last part… ;D

#7

Nice theory, sadly it isn’t true.

Just do a benchmark.

#8

From http://en.wikipedia.org/wiki/IOPS:

[quote]Some HDDs will improve in performance as the number of outstanding IO’s (i.e. queue depth) increases. This is usually the result of more advanced controller logic on the drive performing command queuing and reordering commonly called either Tagged Command Queuing (TCQ) or Native Command Queuing (NCQ). Most commodity SATA drives either cannot do this, or their implementation is so poor that no performance benefit can be seen.[citation needed] Enterprise class SATA drives, such as the Western Digital Raptor and Seagate Barracuda NL will improve by nearly 100% with deep queues.[5] High-end SCSI drives more commonly found in servers, generally show much greater improvement, with the Seagate Savvio exceeding 400 IOPS—more than doubling its performance.[citation needed]
[/quote]
I don’t really know what to make of it. Definitely need a benchmark. Sadly Tom’s Hardware has only been doing SSD benchmarks for the last year. -.-

#9

“reading/writing of packets, adding to queue, that part”

once again, it doesn’t make sense to multithread game logic and single thread the packets reading/writing and adding to queue

should be the other way around

@Conner_

keep the file handler single threaded

and I’m not sure what you mean by “You said it would be bad for networking packet I/O”

do you mean I said it’s bad to single thread it?

it is not as good to have single threaded as multithreaded, but you might not necessarily need to multithread since I don’t think minecraft gets a lot of players per server anyways (???) i was mostly just pointing out that you had the concepts reversed. game logic should be single threaded, something like reading/writing from a network and adding to different queues can run in parallel

#10

Even for small files, it doesn’t make sense to force the disk to seek all over the place when it can just write contiguous blocks of memory one after the other.

#11 
import java.io.BufferedOutputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.LinkedList;
import java.util.Queue;
import java.util.Random;
import java.util.concurrent.CountDownLatch;


public class Main {

	public static final int count = 1000;
	public static final int fileSize = 65536;
	public static CountDownLatch latch;
	public static final Queue<byte[]> queue = new LinkedList<byte[]>();

	public static void main(String[] args) throws InterruptedException {
		Random random = new Random();
		for (int i = 0; i < count; i++) {
			byte[] data = new byte[fileSize];
			random.nextBytes(data);
			queue.add(data);
		}

		long start = System.nanoTime();
		test2();
		long end = System.nanoTime();
		System.out.println((end - start) / 1000000D);
	}

	public static final void test1() throws InterruptedException {
		latch = new CountDownLatch(2);
		new Worker().start();
		new Worker().start();
		latch.await();
	}

	public static final void test2() throws InterruptedException {
		latch = new CountDownLatch(1);
		new Worker().start();
		latch.await();
	}

	static class Worker extends Thread {

		@Override
		public void run() {
			int num = -1;
			byte[] data = null;
			while (true) {
				synchronized (queue) {
					if (queue.isEmpty()) {
						latch.countDown();
						break;
					}
					num = queue.size();
					data = queue.poll();
				}
				try {
					BufferedOutputStream out = new BufferedOutputStream(new FileOutputStream(String.valueOf(num)));
					out.write(data);
					out.close();
				} catch (IOException e) {
					e.printStackTrace();
				}
			}
		}

	}

}

run test 1 then run test 2 (separate instances of JVM).

EDIT: sometimes faster, sometimes slower, the fastest result of test 1 is faster than the fastest result of test 2

#12

If it’s “sometimes faster, sometimes slower” that’s generally a surefire sign that there is something wrong with the benchmark. :-\

#13

depends on my current disk use, not my fault :expressionless: generally speaking though, if you take the fastest from each, in that case multi threaded was faster

#14

TheAgentD - Riven: 0.000001 - 0!!! Marginal semi-unconfirmable victory! (I’m making up words! =D)

#15

I will provide a benchmark later (in a few days).

There is a lot that can go wrong in file-benchmarks, like the OS caching the files in RAM.

I’m mainly talking about large file I/O though, as with tiny files, you’re moving the head in the HDD all over the place anyway.

#16

Yes, large files would indeed not benefit from multi-threading.
Just a small off topic question: Are you going to sue? http://na.leagueoflegends.com/news/new-champion-approaches-riven-exile

#17

btw, overall it’s probably better to have a single thread (just giving an example where multi threaded was -slightly- faster), and we don’t know how big conner’s files are

#18

Only things being written to or read from are Minecraft save files (small data region files and one ~2mb level save) and player saves. Player saves can’t go above like 8kb, depends on the amount of items they have in their inventory.

#19

Now that I think about it, you will always have performance gain from multithreading your file i/o but the question is, is it necessary (probably not lol, unless you’re constantly saving/loading files, like a huge webserver might do)

reading from hdd is synchronous so that won’t happen, requests to it are queued

#20

It doesn’t matter how you ‘reason’ your way out, it’s simply much faster to write large amounts of contiguous bytes, one file at a time.

A HDD doesn’t work like your average I/O queue, where the throughput is constant. The more time the (mechanical) head is moving (due to seeking), the more throughput you lose.

If you write a big file, the head hardly moves.
If you write two a big files, the head moves back and forth on each write (lets assume every 4K).

Every seek takes about 5-10ms, which limits your seeks to 100-200 per second (IOPS).
100 * 4K = 400K / sec
200 * 4K = 800K / sec
This is worst-case scenario, mind you, but the more files you concurrently read/write, the more likely it becomes that you reach that dreadful performance.

This is also why copying a directory-structure with lots of tiny files has such horrible performance: every dozen reads are followed by a seek and the writes are also all over the place…