Notification system created!
It created some fun challenges to solve.
Server-side, we don’t want to create contention/load on the producers that want to create notifications. This means, for example, we don’t want to handle the fanout when we actually create the notification itself.
Instead, we fire off the notifications right away into a queue. Currently, since the server runs on one big machine it gets put in a ConcurrentLinkedQueue.
A separate thread periodically goes through and 1. Pops items out of the queue and assigns them to a user and 2. Cleans up old notifications assigned to a user that were never consumed.
The client-side event loop on the server (which runs on multiple threads) periodically polls notifications for that particular user and pushes them to the client.
This gives us clear separation of responsibilities, and if we need to ever scale horizontally, this architecture lets us.
The impl. is fairly simple:
// BEGIN MANY THREADS
public void addNotification(ID userId, Notification notification) {
notificationsInQueue.add(new NotificationQueued(userId, notification));
}
// If the server shuts off during the call of this method, some notifications will be lost.
// It would be nice if the client could say what notifications it got, and *then* we remove them from the server.
public void getUserNotifications(ID userId, List<Notification> result) {
Queue<Notification> notifications = notificationsByUserId.get(userId);
while (notifications != null && !notifications.isEmpty()) {
Notification next = notifications.poll();
if (next != null) {
result.add(next);
}
}
}
// END MANY THREADS
// BEGIN PROCESSOR THREAD
private void bucketQueueByUser() {
while (!notificationsInQueue.isEmpty()) {
NotificationQueued notificationQueued = notificationsInQueue.poll();
if (notificationQueued != null) {
Queue<Notification> notifications = notificationsByUserId.get(notificationQueued.userId);
if (notifications == null) {
notifications = new ConcurrentLinkedQueue<>();
notifications.add(notificationQueued.notification);
notificationsByUserId.put(notificationQueued.userId, notifications);
} else {
notifications.add(notificationQueued.notification);
}
}
}
}
private void cleanupOldNotifications() {
long now = System.currentTimeMillis();
for (Map.Entry<ID, Queue<Notification>> entry : notificationsByUserId.entrySet()) {
entry.getValue().removeIf((notification) -> notification.createdAt + notification.lifeDuration >= now);
}
}
public void processQueue() {
cleanupOldNotifications();
bucketQueueByUser();
}
// END PROCESSOR THREAD
We also have monitoring for if things like the processor thread taking too long.
The notifications are also persisted to disk for reliability etc. If you do something, like create a very long link, and then close the app, and then we do a deployment (restart server), and come back a week later, you’ll get the notification.
On the client, the notification UI framework is created in libgdx’s regular ol’ scene2d.
We now have notifications for when towers created, links are created, creeps start spawning for new links (attacks/trade routes).
Here’s a quick demo of the result!
https://www.youtube.com/watch?v=CyCDWZgmUpc
I have some plans to expand notifications, maybe after launch, where each notification can be tied to a geographical position, so you can click “see details” on the notification and jump there.
