My recent encounter with UDP packet drop on freebsd boxes gave birth to this post. Here what I learn from that skirmish.
Why UDP needs tune up as compare to TCP ?
Operating system automatically assign TCP buffer space on based on different factors, and TCP sending kernel can monitor buffer flow adjust sending size of data with help of window scaling. These kind of facility is not availble for UDP.
Buffer space for any UDP connection on freebsd is affected by following parameters
net.inet.udp.recvspace – When you open any UDP socket this parameters decides default receiving buffer space for userland data for that socket . You can override that size with help of setsockopt in your code.
kern.ipc.maxsockbuf – The buffer space for whole socket is determine by this parameter. So if you try to open socket with large send and receive buffer and you get error like ‘no buffer space availble’ then you should consider tweaking kern.ipc.maxsockbuf. Sometime you see frequent UDP drops while dealing with large number of tiny UDP packets. Even if your recvspace buffer is not filled up completely , still you will drop the packets. After digging around for a while we figured out that it was happening because we were hitting another hard limit of sockbuf->sb_mbmax, it specifies maximum number of mbufs allocated for each socket. You can increase that limit by increasing kern.ipc.maxsockbuf . You need to restart related services to get advantage of increased
kern.ipc.nmbcluster – this parameter governs ultimate amount memory you have to allocate all sockets opened on that system. This number tells how many number of mbuf cluster should be allocated. Usually each cluster is of 2k size. For example , if you are planning to open 1000 sockets with each having 8k sending and 8k size receving buffer . In this case each socket will need 16k of memory and in total you will need 16M (16k x 1000 ) of memory to handle all 1000 connections.
