Since I’ve been developing online games since 1991, many folks have asked me to recommend a 3rd-party network library for their game project.
Unfortunately, I can’t! At least I can’t recommend one from personal experience because the projects I’ve worked on — Warcraft, Diablo, Starcraft, battle.net and Guild Wars – were all built on proprietary, “from the ground up” network code I developed or co-developed.
But I’ve been meaning to do some research into 3rd-party network libraries so I don’t have to write something from scratch next time; it just takes too long. The core networking libraries I wrote for Guild Wars comprise about 50K lines of code and took several years to fully stabilize because of esoteric issues. To give you a brief idea of some of the scary bits:
- Setting TCP window size: enabling high throughput over long-distance connections without exposing servers to distributed denial-of-service (DDoS) attacks from clients connected to the same server.
- Message buffering: don’t drop a connection just because the other side stalled, but be careful not to allow DDoS attacks.
- Socket shutdown issues: close sockets “nicely” to avoid disconnect delay but don’t expose servers to DDoS attacks.
- Rate-limiting connections & throughput: ensure that servers can’t be abused by DDoS attacks – are you noticing a pattern here?!?
It is difficult to select a network library because there are so many parameters. Are you looking for a reliable message-pusher? Should it use TCP, UDP or both? Do you want object synchronization? Should object synchronization include client-side prediction? Do you want only client-to-server, or do you also need to handle server-to-server communications too? Cryptography? Network firewall traversal (aka UDP hole-punching)?
I’m going to start by eliminating libraries that don’t have a cross-platform story; let’s choose something that works for PC, console and mobile. While game projects don’t need to run on all of them, it is desirable for the programming team to be able to switch to different platforms without having to relearn everything.
Further, using a high-level API is a good choice for most game teams. Sending packets isn’t hard; sending reliable, compressed, scalable, encrypted, low-latency object-synchronization messages is hard. Let someone else do the work!
These high-level, cross-platform network libraries all look like they’re appropriate for game development:
Here are some other high-level libraries I looked at but don’t think are appropriate:
- Torque Network Library (TNL) – no longer supported.
- TNL2 – also appears dead; GPL license prevents use in commercial game clients.
- ENet – lacking high-level documentation.
- ACE – looks dauntingly complex.
And here are some lower-level or more purpose-specific libraries:
- ZeroMQ – great for server-to-server but I wouldn’t use it for client->server: not robust against hacking.
- Boost ASIO – solid and complete, but (flame) using boost is a religious issue for some because it is especially hard-to-read C++ code, and can dramatically increase build-time (/flame).
- LibEvent – doesn’t support Windows well (only uses select, not IOCP).
- LibEv – doesn’t support Windows
- Google Protocol Buffers
- Apache Thrift
So that’s a start on the choosing anyway; I hope that helps reduce the scope of research required to get you started on your next game. Now can I join your beta?
About Patrick Wyatt
You should look into libuv, it uses IOCP on windows and libev on linux: https://github.com/joyent/libuv/
Thanks for the info; I’ve got it on my watch list in github now and will check it out.
fyi, libuv is the library on which node.js is build, so should be solid.
RakNet appears to be targeted at games more than the others and is packaged with Unity. ICE has a nice whitepaper on speed comparisons against industry standards (WCF and RMI):
http://zeroc.com/articles/IcePerformanceWhitePaper.pdf
For serialization,
http://msgpack.org/ has been the only real option for us in Java for real-time multiplayer games. Might be nice in other languages too.
very useful thanks.
I apologize for the offtopic comment, however…
> GPL license prevents use in commercial game clients.
FYI, GPL doesn’t prevent commercial use; it’s actually encouraged.
GPL is often criticized that it doesn’t work for commercial software, as anyone can give it away for free. Well, if you have few gigabytes of art assets that you sell with your game then you don’t have this problem. You can keep them proprietary and still get paid. I really wish more people would notice this option – everyone thinks that you need to make everything free, and this is simply not true.
To be fair, for some games, e.g. MMOs, GPL’ing the engine might not be a good idea, but for your garden variety single player game it’s a win-win situation for everyone involved.
I suspect you’re not in the industry; it’s rarely a possibility to release your source if it contains propriety code from other companies like FMOD or Epic. Even if that’s not the case, it’s still not practical – no matter what the OS community has to say about it.
> it’s rarely a possibility to release your source if it contains propriety code from other companies
If it contains, yes, but if it only uses a proprietary library or two (e.g. MSS or FMOD) then I see no problem.
> Even if that’s not the case, it’s still not practical – no matter what the OS community has to say about it.
Of course it is practical – the thing is, it hasn’t really been tried before, so everyone assumes it won’t work.
For a taste of what it might be like, may I direct to you to the MCP project that grants you partial access to Minecraft’s source code, thanks to which we have things like the Bukkit server and a gazillion of mods, which end up making the game a lot more attractive and contribute to the six million and counting copies sold counter? Without the source access these wouldn’t exists, and Minecraft wouldn’t be what it is today. (Of course in Minecraft’s case releasing the source code as free software wouldn’t work as the game has virtually no art assets, but if you have ten gigabytes of them then it isn’t a problem.)
Assuming you don’t have any dependencies that would make the release completely unpractical (e.g. Unreal Engine dependency), what do you have to “protect” by making a game closed source? Do you fear the pirates? (Hint: that doesn’t stop them; just visit thepiratebay and see for yourself.) Do you fear that someone will steal it? (I doubt anyone would be so stupid.) What exactly could you possibly lose? (Again, assuming we are talking about a non-MMO game without any dependencies on big, proprietary, irreplaceable components and with a nontrivial amount of art assets.)
Hmmm… not sure why you’re experiencing comment length limits; some of my comments are *huge*.
Doesn’t really matter if *you* see the problem. The problem exists. There are several things you can do to link GPLed code with closed source but they come with associated cost.
1. If GPLed code is in its own binary (e.g. DLL) and is spawned by the main process as one-off (load, execute some task, free), then you’re OK. This is not true for networking library.
2. If your closed source is bound with GPL code (they exchange data and call each other) then you have to be the author of GPLed code and provide it with the exception for linking with the closed code solution. Unlikely scenario.
3. If GPLed code is a “system library” (license describes what this means and varies between GPL2 and GPL3) then you can link your closed code with it.
In other words: unless you’re releasing your game’s source, GPLed networking libraries are a no-no.
There is a trend in the game industry to handle too much in the network layer.
Outside of core stuff like proper write/disconnect handling, back pressure handling, etc.., the tcp client should be pretty dumb. Almost ANY client will work if you aren’t trying to do stuff you shouldn’t be doing at the network layer.
There is also a lot of empirical evidence that udp is better for all types of multiplayer games. Many times developers think they need reliable or ordered messaging when they wouldn’t with the right architecture. And if you really need reliable ordered messaging, take a look at UDT, which is supported by at least Raknet and probably one or two others, and doesn’t suffer from some of the built in performance issues that tcp has.
Raknet is probably one of the better tested libraries, although anyone still writing server side code in C++ needs to just stop.
evented io is a lot more important server side, on the client it’s not giving you much.
And protocol buffers are great for games. A huge advantage with them is that they are naturally sparse, they only contain data from fields you populate, and don’t carry any schema info in the message. And the chances that your custom binary format is better? Ya good luck with that.
Zeromq is a pain to work with, terrible error handling. You will end up writing a lot of layers on top of it.
And a little outside the scope of this article, but if you want a server framework that has built in networking and a solid foundation for message passing type architectures, take a look at Akka.
This is more like list, not recommendation or review. I still doesn’t understand what to choose? Do I will still need to write DDoS proctection myself? I think library should take low-level DDoS protection and I take high-level DDoS protection.
ddos are fought through application firewalls anyway
Did you take a look at Kryonet? An often recommended (especially for the very popular LibGDX) Java networking lib.
Regarding TNL2: “GPL license prevents use in commercial game clients.”
Wouldn’t this also apply to ICE? Check out their homepage:
By linking with Ice (directly or indirectly, statically or dynamically,
at compile time or run time), your application is subject to the terms
of the GPL, which require that you release the source code of your
application if and when you “distribute” it.
ICE has a commercial license for propietary products.
Ok, so this is a relatively old article. Still, a good and valuable read nevertheless. Networking is one of the biggest hassle is games nowadays (especially since most gamers now prefer to play a multiplayer game over a strictly singleplayer one).
Have you heard about the lidgren library? If yes, do you have any opinions on it (though AFAIK it’s a .NET/Mono only lib)?
I have heard of the Lidgren library, and in fact I am using it for a game called Moonrise (http://moonrisegame.com) that’s currently in Early Access on Steam.
Lidgren was designed to be used for Peer-to-Peer games (~8 connections), so it’s necessary to extensively refactor the library to make it more effective for Client-to-Server games (~thousands of connections).
Things that I rewrote in my fork of Lidgren:
1. Use O(log N) priority queue for scheduling instead of O(N) array iteration, which is not important when you have 4 connections, but much more so when you’re aiming for 5000+.
2. Remove the send/receive bit-manipulation code and just send bytes. The code spent too much time bit-shifting to achieve maximum packing efficiency.
3. Too much memory consumption — reduce the number of channels.
4. When I forked the code the crypto was broken so I had to rewrite it. I know the crypto code has also been rewritten by Michael Lidgren but I haven’t reviewed those rewrites. Boy, let’s hope my fixes aren’t broken too…
5. I rewrote buffer recycling; it did linear searches through linked lists looking for the right size buffer. This may also have been rewritten by Michael; again, haven’t reviewed his changes.
6. Fixed a bunch of bugs, most of which Michael integrated back into the code base last year.
All that being said, it was very useful to have working code on day 1 of the project so that the rest of the team could be up-and-running, and I could refactor the library to suit our needs over time.
With RakNet having been abandoned with major issues, there doesn’t seem to be any good real-time network libraries (UDP-based) anymore. I’ve been using ENet and it struggles with 1500+ clients. Lidgren being in C# makes me question its speed (at least shouldn’t be as fast as a good C++ library). After having profiled ENet, what you’ve just explained about its performance issues seems to be the same kind of issues ENet is suffering from. ENet was also built for small games (max ~64 players?).
I might have a go at some point at improving ENet to run with more clients by using similar methods that you used, so thanks a lot for sharing!
Hi Patrick, I am using Lidgren library for my game and have some trouble with memory cosumption. Could you share your fixed-bug code ?
Great blog Patric
Hey i published few tutorials about building MMO’s
for begginers please check it out
This one is usnig libuv & libwebsocktes & cocos2d-x-js
http://www.gamedevcraft.com/2016/08/part-1-multiplayer-websocket-game.html