est's blog

开篇帖子侃一下HAProxy这个神器有什么神器之处。

写过proxy的人都会写类似的语句：

localstream.write(upstream.read(1024))
upstream.write(localstream.read(1024))

其中1024为业界良心艹榴网的chunk或者一个buffer size。这样的操作在各种router, proxy, firewall 里太常见了。

那么HAProxy有什么特别的地方呢？秘密就是，上游socket的接受buffer和本地socket的发送buffer为同一块内存。也就是说，上游socket数据来了，本地socket自动发送这一块buffer的数据。效率真是高爆了。这块buffer数据还可以用内核调用splice() 和 tee() 来达到分流和分离信号的目的。

btw sendfile() 是不行的，源必须是可以mmap的buffer，目标必须是个socket。

那么弱爆了的python可否跪舔这种高级特性呢？2.7和3.X里是可以的，因为现在PEP 3118统一了各种streaming object 的buffer protocol。只需要用一个叫memoryview的引用 就可以实现共享buffer而达到zero-copy的目的了。

C API见这里，python代码例子在这里，具体核心调用就是 file.readinto() 和 socket.recv_into() 方法。具体代码留作读者课后练习。

It is a bit different than your standard FEC. Traditionally, FEC works on a block (of bits/bytes) by block basis. You need to the entire block to decode the message. So the transmitter needs a good guess on the channel capacity to choose the correct coding strength for the block. If the sender guesses wrong, the channel is either under utilized (sending too slow) or over utilized (lost packet). In practice it is hard to guess the channel state.
The novelty is their work is the use of network coding, a form of rateless coding. A rateless code does not operate on a block by block basis. Instead once enough bits are received, the message can be decoded. The rate automatically adapts to the channel’s capacity because of the encoding/decoding of the code.
Rateless codes are also known by different names, e.g. fountain (LT) codes, raptor codes, tornado codes, hybrid-ARQ, strider codes, spinal codes, and in a sense network coding (though this may have more applications).

最近一篇paper在网上吵得很火，这段话基本总结了这个技术的思想。

纸：http://www.mit.edu/~medard/papers2011/Modeling%20Network%20Coded%20TCP.pdf

from extremetech via 1, 2, 3

安装pv：
sudo apt-get install pv

下行：
$ ssh est@my_host 'cat /dev/zero' | pv > /dev/null
144kB 0:00:05 [ 121kB/s] [ <=> ]

上行：
$ yes | pv | ssh est@my_host "cat > /dev/null"
2.06MB 0:00:05 [1.96MB/s] [ <=> ]

测试本机IO能力：
$ pv /dev/zero > /dev/null
75.3GB 0:00:09 [8.56GB/s] [ <=> ]

其他技巧：
http://blog.urfix.com/9-tricks-pv-pipe-viewer/

恩。真好玩。我发现办公室的商用ADSL对称上下行就是屌啊。2MB上传速度我擦不用来挂PT可惜了。