Scaling memcached at Facebook
Facebook Engineering
If you've read anything about scaling large websites, you've probably heard about memcached. memcached is a high-performance, distributed memory object caching system. Here at Facebook, we're likely the world's largest user of memcached. We use memcached to alleviate database load. memcached is already fast, but we need it to be faster and more efficient than most installations. We use more than 800 servers supplying over 28 terabytes of memory to our users. Over the past year as Facebook's popularity has skyrocketed, we've run into a number of scaling issues. This ever increasing demand has required us to make modifications to both our operating system and memcached to achieve the performance that provides the best possible experience for our users.
Because we have thousands and thousands of computers, each running a hundred or more Apache processes, we end up with hundreds of thousands of TCP connections open to our memcached processes. The connections themselves are not a big problem, but the way memcached allocates memory for each TCP connection is. memcached uses a per-connection buffer to read and write data out over the network. When you get into hundreds of thousands of connections, this adds up to gigabytes of memory-- memory that could be better used to store user data. To reclaim this memory for user data, we implemented a per-thread shared connection buffer pool for TCP and UDP sockets. This change enabled us to reclaim multiple gigabytes of memory per server.
Although we improved the memory efficiency with TCP, we moved to UDP for get operations to reduce network traffic and implement application-level flow control for multi-gets (gets of hundreds of keys in parallel). We discovered that under load on Linux, UDP performance was downright horrible. This is caused by considerable lock contention on the UDP socket lock when transmitting through a single socket from multiple threads. Fixing the kernel by breaking up the lock is not easy. Instead, we used separate UDP sockets for transmitting replies (with one of these reply sockets per thread). With this change, we were able to deploy UDP without compromising performance on the backend.
Another issue we saw in Linux is that under load, one core would get saturated, doing network soft interrupt handing, throttling network IO. In Linux, a network interrupt is delivered to one of the cores, consequently all receive soft interrupt network processing happens on that one core. Additionally, we saw an excessively high rate of interrupts for certain network cards. We solved both of these by introducing “opportunistic” polling of the network interfaces. In this model, we do a combination of interrupt driven and polling driven network IO. We poll the network interface anytime we enter the network driver (typically for transmitting a packet) and from the process scheduler’s idle loop. In addition, we also take interrupts (to keep latencies bounded) but we take far fewer network interrupts (typically by setting interrupt coalescing thresholds aggressively). Since we do network transmission on every core and since we poll for network IO from the scheduler’s idle loop, we distribute network processing evenly across all cores.
Finally, as we started deploying 8-core machines and in our testing, we discovered new bottlenecks. First, memcached's stat collection relied on a global lock. A nuisance with 4 cores, with 8 cores, the lock now accounted for 20-30% of CPU usage. We eliminated this bottleneck by moving stats collection per-thread and aggregating results on-demand. Second, we noticed that as we increased the number of threads transmitting UDP packets, performance decreased. We found significant contention on the lock that protects each network device’s transmit queue. Packets are enqueued for transmission and dequeued by the device driver. This queue is managed bv Linux’s “netdevice” layer that sits in-between IP and device drivers. Packets are added and removed from the queue one at a time, causing significant contention. One of our engineers changed the dequeue algorithm to batch dequeues for transmit, drop the queue lock, and then transmit the batched packets. This change amortizes the cost of the lock acquisition over many packets and reduces lock contention significantly, allowing us to scale memcached to 8 threads on an 8-core system.
Since we’ve made all these changes, we have been able to scale memcached to handle 200,000 UDP requests per second with an average latency of 173 microseconds. The total throughput achieved is 300,000 UDP requests/s, but the latency at that request rate is too high to be useful in our system. This is an amazing increase from 50,000 UDP requests/s using the stock version of Linux and memcached.
We’re hoping to get our changes integrated into the official memcached repository soon, but until that happens, we’ve decided to release all our changes to memcached on github.
Peter VK
Just one question. How do you get from 200 to 500K req/s?
Karah Nazor
geek talk!
Jim Greer
That is very impressive and kind of scary to think about... Kongregate is very happy to be chugging along with I think 4 memcached processes.
Phillip Morelock
Awesome.
Dirk Samwood
Powerful stuff!
Ryan Christensen
Great stuff to learn you are moving more to UDP. I have been stating my case for this for many things I am working on and this helps. Thanks for sharing.
José Celestino
Nice. Great work. 300k req/s is jaw droping.
Neil Bliss
Hear hear for greater untilization of UDP! TCP is unncessarily chatty in many network communication implementations. With today's extremely reliable networks (especially over short distances), UDP is a great way to cut down on network traffic.
Nice work fellows!
Nice work fellows!
Michael McGrew
I would love to hear how you scale MySQL. It can't use more than one core efficiently. Do you use the google SMP patches?
Dan Simoes
Nice work.
David McKay
Lovin it.
Russ Garrett
Easily the best Engineering@Facebook post I've read - this is so useful to us, thanks for the great work on memcached!
Greg Lindahl
Neil, don't write off TCP so quickly. People who reinvent it often find that they need some important feature like congestion control. memcached operations are sometimes pretty big, which might call for clever windowing. So far I'm 1 win - 1 loss for major UDP usage...
Murray Stokely
great stuff. Enjoyed the talk about this last night.
Eric Sabban
We use memcached extensive as well. It has been invaluable for our live data, some of which only exists for 5 seconds at a time (or so).
Database servers would grind to a halt otherwise.
Database servers would grind to a halt otherwise.
Shant Hovsepian
Thought about trying BerkeleyDB HA?
The issues with interrupt driven network devices have been studied extensively in research for example
"Lazy Receiver Processing (LRP): A Network Subsystem Architecture for Server Systems"
"The Click Modular Router"
Seemed like a nice solution you guys have come up with.
The issues with interrupt driven network devices have been studied extensively in research for example
"Lazy Receiver Processing (LRP): A Network Subsystem Architecture for Server Systems"
"The Click Modular Router"
Seemed like a nice solution you guys have come up with.
Bosko Milekic
awesomeness
Troy Thompson
I'm strangely turned on by this. Geek porn?
James Byers
Steve Grimm is my memcached hero.
Siqi Chen
Paul Saab, you are ridiculous.
Sikander Iqbal
>> "more than 800 servers supplying over 28 terabytes of memory"
Wow!
Wow!
Richard Jones
Great post.
How about releasing the PHP client you use too? :)
How about releasing the PHP client you use too? :)
Nguyễn Đăng Ngọc
Great post. Thank you!
David Precious
Excellent work guys, and great to see that it's being released back to the community too, so everyone can benefit from your improvements! That's the spirit of Open Source right there.
/me raises a glass of Christmas cheer to the FB developers :)
/me raises a glass of Christmas cheer to the FB developers :)
Fabio Varesano
Great work guys.. I hope you will share back with the Open Source community the improvements you made.
Nguyễn Đăng Ngọc
I installed memcached from danga success. But I got errors when I install from this source (github). Errors msg:
shell$ cd src
shell$ autogen.sh
shell$ ./configure --enable-threads
shell$ make
shell$ make install
.....
cc1: warnings being treated as errors
sigseg.c: In function âsignal_segvâ:
sigseg.c:66: warning: format â%02luâ expects type âlong unsigned intâ, but argument 3 has type âsize_tâ
make[2]: *** [memcached-sigseg.o] Error 1
make[2]: Leaving directory `/root/fbmarc-facebook-memcached-702558756225f784a8b27412e94656cd6448931b/src'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/fbmarc-facebook-memcached-702558756225f784a8b27412e94656cd6448931b/src'
make: *** [all] Error 2
OS: CentOS 5.1, libevent 1.4.8 stable
shell$ cd src
shell$ autogen.sh
shell$ ./configure --enable-threads
shell$ make
shell$ make install
.....
cc1: warnings being treated as errors
sigseg.c: In function âsignal_segvâ:
sigseg.c:66: warning: format â%02luâ expects type âlong unsigned intâ, but argument 3 has type âsize_tâ
make[2]: *** [memcached-sigseg.o] Error 1
make[2]: Leaving directory `/root/fbmarc-facebook-memcached-702558756225f784a8b27412e94656cd6448931b/src'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/fbmarc-facebook-memcached-702558756225f784a8b27412e94656cd6448931b/src'
make: *** [all] Error 2
OS: CentOS 5.1, libevent 1.4.8 stable
Abbas Ahmed
Hmmm... I wonder if all these super tweaks have found their way to the memcached available on the Danga.com website??
Great Work FB Engineers...
Great Work FB Engineers...
Brendan O'Connor
Paul, that is one hot post.
Brendan O'Connor
and when i was reading the parts about the linux kernel, in my head i was hearing you say "but if only we used freebsd!" :)
Fabrizio Codello
I've always wondered what were the "traffic stats" behind facebook =)
Carlos Oliveira
This is a great job, congratulations. I would be interested in knowing what tools you use to collect data about network throughput.
Wilfried Schobeiri
/me watches as calamity approaches via mainline merge request
Vijay Gill
tcp is unnecessarily chatty? I am badly confused.
Nicholas Smit
Well done guys - excellent work, and thanks for releasing this to the community.
Alexander Mahabir
Here comes the crash!!! j/k ;)
nice one!
nice one!
Anthony David Adams
Very impressed with 300k. Looking forward to more advances.
Philip Jacob
Is that 200k requests per node or in total for all your 800 nodes?
Ruturaj Vartak
one of the blogs, that post less crap/daily talk and more of the geek talk
Hugues Le Bars
As usual Paul, you rock !!
Let me test this :)
Let me test this :)
Luke Lu
Interesting hacks (are you using RSS (not that one) and MSI-X or not?) I wonder what's the messages size used to achieve 200k qps (assuming it's per node, of course, otherwise it's rather slow for 800 machines) and the bandwidth usage at this throughput. For people storing average 1KB data per request, you only need about 100k qps to max out a 1 GigE NIC. You'll need a monster switch (or extremely cutting edge meshed switches) to fully utilize the 800 machine memcached boxes.
This post seems to reveal some fundamental architecture problems of facebook (?)
This post seems to reveal some fundamental architecture problems of facebook (?)
Paul Saab
The 200,000 qps number comes from a single node.
Across our entire tier we're doing 50M keys/s.
As for MSI-X and RSS, it is something we're investigating, but we had to make our current hardware work which does not have these features.
Across our entire tier we're doing 50M keys/s.
As for MSI-X and RSS, it is something we're investigating, but we had to make our current hardware work which does not have these features.
Jian Yao
Great! We are also interested in your memcached php client. Without it, we can not utilize the best part of your work - memached UDP support
Cao Li
Here is the Chinese translation of this article:
这里是该文章的中文翻译：
http://shiningray.cn/scaling-memcached-at-facebook.html
这里是该文章的中文翻译：
http://shiningray.cn/scaling-memcached-at-facebook.html
Cao Li
I have a problem on translation
"under load" appears several times in the article.
is it "heavy load" or "light load"?
"under load" appears several times in the article.
is it "heavy load" or "light load"?
Matias Estrada
> "under load" appears several times in the article.
> is it "heavy load" or "light load"?
high load
> is it "heavy load" or "light load"?
high load
Hussain Ali
Interesting you mention linux, I remember reading Facebook was using FreeBSD 6.2 64 bit dual dual core amd's. Do you have any benchmarks against both linux & FreeBSD ? Would be interesting to see how the scaling wars, espically with the newer libthr/jemalloc/google malloc turn code out.
Chika Tambun
FaceBook(FB) servers relies on Micro$oft(M$) product, doesn't they?
^^
^^
Chen Maosheng
In Linux, a network interrupt is delivered to one of the cores, consequently all receive soft interrupt network processing happens on that one core.
-----------------------------------------------------------------------------------------
How to solve this problem or compensate for it? Thanks for your awesome stuff.
-----------------------------------------------------------------------------------------
How to solve this problem or compensate for it? Thanks for your awesome stuff.
Jon Pincus
very interesting post, thanks for taking the time ...
Jay Faulkner
Nice to see other people experience the softirq problem. We're attacking it from every angle, but it's a very difficult problem to solve.
Stephane Tsacas
What I understand from what it is written is "memcached is poorly designed".
And its corollary : linux is lame, too.
(feel free to flame me ;-) ).
And its corollary : linux is lame, too.
(feel free to flame me ;-) ).
Tony Juays
Now that's what I am talking about!
Dag Wieers
After reading this post, I am concerned that the fixes are being implemented for a single set of problems. Usually that is not the way it works inside the Open Source community, issues and changes need to be discussed and designed with other user's requirements and developer's wishes in mind as well.
I hope I am wrong, but I fear that releasing a big patch like this (which is legally required) might not fare well to get the changes applied upstream. And that may be hurting your goals in the long run as well.
For one, if there are UDP performance problems working around them in memcached will likely not be accepted. The real fix is in the kernel. This may be practically the easiest fix, but it is not the cleanest and may cause other problems elsewhere that are harder to find and fix. Additionally, everyone else is having those UDP performance problems on Linux.
Again, I hope I am wrong about this.
I hope I am wrong, but I fear that releasing a big patch like this (which is legally required) might not fare well to get the changes applied upstream. And that may be hurting your goals in the long run as well.
For one, if there are UDP performance problems working around them in memcached will likely not be accepted. The real fix is in the kernel. This may be practically the easiest fix, but it is not the cleanest and may cause other problems elsewhere that are harder to find and fix. Additionally, everyone else is having those UDP performance problems on Linux.
Again, I hope I am wrong about this.
Jay Faulkner
I agree with Dag. I hate that the fix you put in here for the problem is app specific.
There's a lot of people (me included) who have trouble with softirqs on load balancers, firewalls, and other things that push traffic. It'd be awesome if you guys would coordinate with people upstream to get your fixes pushed down.
There's a lot of people (me included) who have trouble with softirqs on load balancers, firewalls, and other things that push traffic. It'd be awesome if you guys would coordinate with people upstream to get your fixes pushed down.
Alex Valentine
This note is now linked on slashdot, but I think facebook can handle it.
Daniel Nishimura
When you moved from TCP to UDP, how do you ensure reliable delivery of the data? I had a similar situation where we switched from TCP to UDP for my IPC implementation. However, I encountered some data loss even though the two communicating processes were on the same physical node. Did you implement your own transport mechanism?
Christopher Tai
This is badass.
Keith Barrette
I DISAGREE with Dag - there's nothing saying these changes HAVE to be integrated into the main memcached line. IMO there's nothing wrong with customizing a tool to fit your app-specific needs. If the community at large decides they want this, so be it, if not, I'm sure facebook is happy with their performance increase.
Although I have to wonder - after all this, what's really left of memcached? There's not a whole lot there to begin with :)
Although I have to wonder - after all this, what's really left of memcached? There's not a whole lot there to begin with :)
Brian Klug
Wonder how much your servers are getting pounded right now since you're slashdotted: http://developers.slashdot.org/article.pl?sid=08%2F12%2F17%2F1443216&from=rss
Mike Andrews
I seem to remember FreeBSD just recently did some work on UDP lock contention in the kernel that improved bind9's performance. I think it made it into 7.x. MSI-X works pretty well too.
Still though, I hadn't considered UDP-based memcached until now... mostly because we're using persistent TCP connections (mod_perl holds 'em open) so we don't have the handshake overhead per lookup. In theory that should give comparable network overhead to UDP, though I haven't checked it w/ tcpdump. On the other hand our server farm is under 10 machines, nowhere near 800 :)
Still though, I hadn't considered UDP-based memcached until now... mostly because we're using persistent TCP connections (mod_perl holds 'em open) so we don't have the handshake overhead per lookup. In theory that should give comparable network overhead to UDP, though I haven't checked it w/ tcpdump. On the other hand our server farm is under 10 machines, nowhere near 800 :)
Paul Saab
The problem with TCP is if you drop a packet in your network, and you're not constantly transmitting from the same client socket, the OS will not notice for 250ms. This is why we use UDP. We have our own php client that does all this magic for us, and we'll be releasing that in the coming months.
As far as FreeBSD is concerned, it's still got a ways to get the latency down. Some of the work that is going on in CURRENT will definitely allow for FreeBSD to compete without needing any external modifications.
As far as FreeBSD is concerned, it's still got a ways to get the latency down. Some of the work that is going on in CURRENT will definitely allow for FreeBSD to compete without needing any external modifications.
Michael Hardman
I'm so jealous...
Alex Kompel
Great effort but you still are trying to scale memcached vertically which is the battle you are ultimately going to lose. Perhaps you should also look into partitioning caches. Connection pooling on the host level will also help (perhaps even serializing memcached requests from apache at the host level into a single TCP connection)
Sean Daniel Weeks
200K req/s - wow. Ya, I'm only in the 50 req/s range :-P.
Logan Land
good job FB
Changhao Jiang
This is awesome!
Timmothy Posey
What about cloud/grid computing?
Federico Ceratto
Great job!
Agrawal Mohit
You guys are awesome!! Many thanks.
Mauricio Hernández Durán
Anyone have any more interesting links like this to share?
Gerhard Mack
Are you guys working with the Linux kernel folks to deal with Linux issues you have discovered?
Luke Shepard
Great post Paul. You guys rock.
John Yee
Great job with the optimizing and the explanation. Most importantly, thank you for releasing your hard work for others to use!
Oz Dustie
It sounds like the main cause of doing all these is the high number of connections -- each machine is running hundreds of apache processes, which in turn connect to each memcached instance. I don't know php enough, but I am curious to know if it a php/apache limitation that we can't put just few instances of apache processes to utilize all machine resources? I assume you guys are using (prefork) mpm in apache.
With that many connections, it will take more memory to manange TCP vs UDP. Furthermore, as connections are not shared between processes, it is harder to keep them alive or detect stale connections.
If prefork mpm is required, is it better to introduce mid layer (app layer) to handle connection related challenges, so each process only connection to one guy who proxy to other logic (e.g. memcached) instead of changing everything to UDP and changing buffering of memcached?
Regardless, I have to say this is some amazing hacks, sometimes I'd dream of doing it myself :)
With that many connections, it will take more memory to manange TCP vs UDP. Furthermore, as connections are not shared between processes, it is harder to keep them alive or detect stale connections.
If prefork mpm is required, is it better to introduce mid layer (app layer) to handle connection related challenges, so each process only connection to one guy who proxy to other logic (e.g. memcached) instead of changing everything to UDP and changing buffering of memcached?
Regardless, I have to say this is some amazing hacks, sometimes I'd dream of doing it myself :)
Nathan Boolean Trujillo
it's always very cool to see how the big guys do it, and what's even more cool is that it is open sauce.
kudos.
And yes, I am with Dagas well, but hopefully they will put these tweaks into a configure option, so we have the option of not using them
Now I guess I have to start/join the group of people who are asking why you chose linux over FreeBSD :)
kudos.
And yes, I am with Dagas well, but hopefully they will put these tweaks into a configure option, so we have the option of not using them
Now I guess I have to start/join the group of people who are asking why you chose linux over FreeBSD :)
Manish Shrestha
wow...awesomeness!!!
Numan Harun
wow~~! you are so awesome that I don't really understand most of it but I'm sure this is impressive
Moshe Kaplan
Hi,
Do you use or consider to use Infiniband as backbone solution to solve CPU utilization due to network processing?
Best,
Moshe
Do you use or consider to use Infiniband as backbone solution to solve CPU utilization due to network processing?
Best,
Moshe
Manish Sinha
That's one of the best scalability note I have ever read. Highly informative. Thanks!
Chris Thompson
This amount of caching/traffic is just insane. I would kill to get to work with this stuff.
Tinnycloud Mao
We have great interest in memcached since in our lab we are doing some research on using memcached cache Mysql data, We have do some primary test on the source code,
In fact, we thought the key length has great impact on the throughput, As u mentioned, u multi-get will get hundreds of keys in parallel, as I know, memcached doesn't support command requests on multi-UDP packets, so if a UDP packet is 1400(default is memcached), for 100 keys, the average key length may only about 10bytes. In fact, the requests/second is quite related to the key length, 4 length 100 keys in parallel will have 300K/S, but if 10 length 100 keys will only have 120K/S,
In our test, no modification on kernel level, we are not sure the epoll and the MSI-X has solved interrupt problem
In fact, we thought the key length has great impact on the throughput, As u mentioned, u multi-get will get hundreds of keys in parallel, as I know, memcached doesn't support command requests on multi-UDP packets, so if a UDP packet is 1400(default is memcached), for 100 keys, the average key length may only about 10bytes. In fact, the requests/second is quite related to the key length, 4 length 100 keys in parallel will have 300K/S, but if 10 length 100 keys will only have 120K/S,
In our test, no modification on kernel level, we are not sure the epoll and the MSI-X has solved interrupt problem
Christopher Dale Fogleman
I'm an audio engineer looking for work, I have done this for 20 years and would like to make a living with my degree. I've done many projects from bands including my own music, foley work, raido bits, anything that has sound. If anyone has an opening please contact chrisfogleman@yahoo.com. thank you
Tuhin Barua
how do you distribute keys or partition your data in 800 servers?
do you use any consistent hashing algo like ketama?
cant wait for your php client :)
do you use any consistent hashing algo like ketama?
cant wait for your php client :)
Nonhlanhla Mabilo
How to apply for intern,in electrical engenering
Maarten Troonbeeckx
can anyone tell me how many crew members there are in the "facebook crew"
Malcolm Derksen
wow geek
Trivuz Alam
just great :)
Rayhan Chowdhury
great :)
Facebook is promoting open source. Interested to know what tweak you have applied on MySQL.
Facebook is promoting open source. Interested to know what tweak you have applied on MySQL.
Richard Slade
sweet :)
Lazy Jay
sickkk
Yingkuan Liu
@Cao Li
>>"under load" appears several times in the article.
>>is it "heavy load" or "light load"?
No it's meant "under load" where on a multi-core system one core get saturated serving requests while the others stay idle.
>>"under load" appears several times in the article.
>>is it "heavy load" or "light load"?
No it's meant "under load" where on a multi-core system one core get saturated serving requests while the others stay idle.
Eliot Shepard
Thanks. Where has the repository gone?
Benjamin Herrenschmidt
Hi ! reading this a bit late...
Regarding rx network interrupts, recent kernels now support multiple rx queues. With appropriate network adapters, you can then get a queue + a separate interrupt per core, with the adapter hashing incoming packets to try to keep a given flow onto one core.
Regarding rx network interrupts, recent kernels now support multiple rx queues. With appropriate network adapters, you can then get a queue + a separate interrupt per core, with the adapter hashing incoming packets to try to keep a given flow onto one core.
Matthew Sinclair
also having PHP running hogs a lot of memory so why use it!???
Francesco Vollero
Where we can found the kernel patches?
@Matthew: What language have to use facebook for you?
@Matthew: What language have to use facebook for you?
Jeremy Lemaire
Nice work. Have these changes been integrated into the official memcached repository yet or do I still need to pull them from github?
Mathieu Richardoz
Thank you for sharing, this information should come in handy.
Akande Olajide Kayode
Hello Facebook Great Team,
I really like what you are doing infact your operation make the world more smaller and you have construct a bridge that link everyone and to make far away relatives, friends and family feel more closer. You doing good well done job here.
One thing that I will like you to add again on this newpage is the Recently added and the Alpherbetical order name of friends that you have removed, please if you can add it again on the new page it will be complete.
Thanks
I really like what you are doing infact your operation make the world more smaller and you have construct a bridge that link everyone and to make far away relatives, friends and family feel more closer. You doing good well done job here.
One thing that I will like you to add again on this newpage is the Recently added and the Alpherbetical order name of friends that you have removed, please if you can add it again on the new page it will be complete.
Thanks
Daniel Yuan
batch, reduce lock contenction
Moronkreacionz In
@Paul : the github link is broken
- http://github.com/fbmarc/facebook-memcached
can you provide the most recent github repos for memcached+udp, also where can I find the FB php memcache client ?
~ Vj
- http://github.com/fbmarc/facebook-memcached
can you provide the most recent github repos for memcached+udp, also where can I find the FB php memcache client ?
~ Vj
Moronkreacionz In
Sorry Paul, found the FB memcached+udp link http://github.com/facebook/memcached
Sathiya N Sundarajan
this is very valuable information/experience, to any team building large scale distributed applications.
Wiliam Felisiak Passaglia Castilhos
Impressive, how great is the volume of data! Even on a robust OS like Linux.
徐杭
memcached
David Cho
fantastic!
Rogatz
amazing, facebook type of application always need high performance. it is really nice to see how these projects are ensuring performance for their user.
wish many tips and tricks will come out from these type of projects.
best wishes,
wish many tips and tricks will come out from these type of projects.
best wishes,
Sai Nallapaneni
Loved the article. So many complex problems which need a lot of in depth analysis... kudos to the team.
Sai Nallapaneni
When you were talking about cores being hogged by network interrupts, did you guys try changing CPU affinity? by default, all interrupts go to CPU-0 (core 0), so if you have cpu intensive application, you can force it to run on all cores other than core 0. This itself can give an application unto 20-30% jump.
Adil Ahmed
git page not found :(
Steven Palmer
I cannot find the code you optimized on github
Thomas Beaver
To think I found this while searching for optimization of memcache and NGINX based services. Fantastic information BUT my mind was blown by the amount of engineering and analysis put into various layers of technology to net these massive improvements - especially on the network level. I'd love to employ such techniques on my server but lack the fundamental knowledge of recompiling kernels for improvements in the way TCP/UDP interacts with specific services seems like a daunting task. 8-/
Kevin Watt
Thomas - the same search brought me here too, they must have improved in that search result recently. Epic stuff, probably entirely out of date now that it's 3+ years later, which is also funny to think about.
Caleb Techbay KE
I haven't the slightest idea what this post is talking about because I don't speak Geek! but it sounds smart so I gotta like it
Tonglin Li
Nice work! But one thing is not very clear, when you have high concurrent UDP communication, packets loss occurs often. How do you handle that?
Lampstar Su
Great solution! Extensional, instead of memcached ( network bottlenecks ), How do you think if we can implement memcached + "Local Caching system" ?
Qiu Billow
good jobs
Silviu Sorin
I admin i did all these http request that put this big load on the faceboox servers, because i browse facebook aaal daay looong :)
Deep Parekh
awesome.....
Alain Patrick Ndengera
UDP is faster than TCP. The problem will be lost packets
Sebastian Godelet
@Foo Bar, technical does not mean geek, you ignorant fool
Sunil Patil
We have developed a patch for memcached for "processing multi-get queries efficiently". Have a look at: https://groups.google.com/forum/?fromgroups#!topic/memcached/n8zLJRyYNY4
Bi Xue
173 microseconds is client side measurement or server side?
Joe Yahchouchi
Thank you for the article. Old but still applicable. Memcached is awesome
Pranay Choudhary
Great article. I was just wondering how do you handle hundreds of thousands of HTTP requests per second. Which Load Balancer can take that amount of requests?
Airul
http://masterplans.info
Syed Muhammad
So, Facebook is using Apache?
Rajat Singh
Superb article. But the way you optimized is what makes it incredible!
Engr Nzurumike David Ezenwata
myengineering.com.ng is for students $ professionals of engineering, be it civil, electrical, mechanical, chemical.. for free books, softwares ready for download
join us @ http://www.myengineering.com.ng
join us @ http://www.myengineering.com.ng
Srinivasan
Thank you very much !! Paul Saab
Srinivasan
Paul Saab : How do one find that the Network IO interrupts are handled by a single core and rest gets soft interrupt ? Could you please let me know. Did u guys wrote the interface and logged the core ID. Could you please add the procedure you followed. Thanks in advance
Кирилл Пальцев
Pranay Choudhary well it not a rocket science you use dns round robin balancing (maybe some geo dns stuff) + haproxy or lvs or some hardware solution to distribute requests along first front webservers (nginx, lttpd or your custom if you are facebook)
Puskar Dwivedi
Nice
Alistair Budd
How many packets per second are you up to now per server? And how many NIC's per server, and what is the NIC speed?
Viyond Pay
赞
Akam Omer Faqe Rasul
I love the one who wrote this article
Eswar Yaganti
github source link is down
Marco De Mojà
http://www.talosintelligence.com/reports/TALOS-2016-0219/
Thulasi Ram
Thanks Paul Saab
PK Jajara
Broken GitHub URL: https://github.com/fbmarc/facebook-memcached
Working GitHub URL: https://github.com/fbmarc/facebook-memcached-old
Cheers!
Working GitHub URL: https://github.com/fbmarc/facebook-memcached-old
Cheers!
Khachik Sahakyan
Our hardware accelerated Memcached server with FPGA handles 12.8 Million Requests Per Second guys :)
http://grovf.com
Anyway cool result for the processor!
http://grovf.com
Anyway cool result for the processor!
Badran Mufti Dehan
@[100010210450894:2048:Farman Kodo] @[100001661527294:2048:Ahmad Zewar Umer] @[100026284689158:2048:روار فارس] @[100026067737155:2048:Bayar Hasan] @[100004309239128:2048:Dyar Xoshnaw] @[100033641171215:2048:Hewa Balashawa] @[100034667424309:2048:Shano Kirdi] @[100001233025064:2048:Hoshang B. Al-Assadi] @[100000558864882:2048:Rebaz Rebaz Jaf] @[100000099716260:2048:Shwan A Hassan] @[100002159302941:2048:مراد يشار مشكو] @[100001022620664:2048:Haval Ahmad] @[100010506045104:2048:Ali H Axa] @[100000956380921:2048:Hana Hashim] @[100027908915225:2048:Karwan Sian] @[100024137797312:2048:Kaptn Ziko] @[100022667749800:2048:Gaylan Ardalan] @[100022042171817:2048:Star Ismail] @[100020029846919:2048:ئیبراهیم مەموندی] @[100016955191640:2048:Chaw Rash] @[100016927326551:2048:كارزان سليم] @[100016878766666:2048:Âhmäd Gÿãñ] @[100016246952618:2048:Omer Massifi] @[100015770474999:2048:Rozgar Xalid] @[100015611491166:2048:ته حسين ته ها] @[100015596972300:2048:Mer Dyar] @[100013431634458:2048:کاره کملیات] @[100012847922702:2048:خولی نایابی کوردستان] @[100012621518649:2048:Kako Hawlere]
Chad Little
Good job all!
Deri Herdianto
Ok
