Building Timeline: Scaling up to hold your life story
Timeline isn’t just a bold new look for Facebook­—it’s also the product of a remarkably ambitious engineering effort. While our earlier profile pages surfaced a few days or weeks of activity, from the onset we knew that with Timeline we had to think in terms of years and even decades. At a high level we needed to scan, aggregate, and rank posts, shares, photos and check-ins to surface the most significant events over years of Facebook activity.
 
The schedule for Timeline was very aggressive. When we sat down to build the system, one of our key priorities was eliminating technical risk by keeping the system as simple as possible and relying on internally-proven technologies. After a few discussions we decided to build on four of our core technologies: MySQL/InnoDB  for storage and replication, Multifeed (the technology that powers News Feed) for ranking, Thrift for communications, and memcached for caching. We chose well-understood technologies so we could better predict capacity needs and rely on our existing monitoring and operational tool kits.
 
Denormalizing the data
Before we began Timeline, our existing data was highly normalized, which required many round trips to the databases. Because of this, we relied on caching to keep everything fast. When data wasn’t found in cache, it was unlikely to be clustered together on disk, which led to lots of potentially slow, random disk IO. To support our ranking model for Timeline, we would have had to keep the entire data set in cache, including low-value data that wasn’t displayed.
 
A massive denormalization process was required to ensure all the data necessary for ranking was available in a small number of IO-efficient database requests
 
Once denormalized, each row in the database contained both information about the action and enough ranking metadata so it could be selected or discarded without additional data fetches. Data is now sorted by (user, time) on disk and InnoDB does a great job of streaming data from disk with a primary key range query.
 
Some of our specific challenges of the denormalization process were:
 
1. Dozens of legacy data formats that evolved over years. Peter Ondruška, a Facebook summer intern, defined a custom language to concisely express our data format conversion rules and wrote a compiler to turn this into runnable PHP. Three “data archeologists” wrote the conversion rules.
2. Non-recent activity data had been moved to slow network storage. We hacked a read-only build of MySQL and deployed hundreds of servers to exert maximum IO pressure and copy this data out in weeks instead of months.
3. Massive join queries that did tons of random IO. We consolidated join tables into a tier of flash-only databases. Traditionally PHP can perform database queries on only one server at a time, so we wrote a parallelizing query proxy that allowed us to query the entire join tier in parallel.
4. Future-proofing the data schema. We adopted a data model that’s compatible with Multifeed. It’s more flexible and provides more semantic context around data with the added benefit of allowing more code reuse.
 
Timeline aggregator 
We built the Timeline aggregator on top of the database. It started its life as a modified version of the Multifeed Aggregator that powers News Feed, but now it runs locally on each database box, allowing us to max out the disks without sending any data over the network that won’t be displayed on the page.
 
The aggregator provides a set of story generators that handle everything from geographically clustering nearby check-ins to ranking status updates. These generators are implemented in C++ and can run all these analyses in a few milliseconds, much faster than PHP could. Much of the generator logic is decomposed into a sequence of simple operations that can be reused to write new generators with minimal effort.
 
Caching is an important part of any Facebook project. One of the nice properties of Timeline is that the results of big queries, such as ranking all your activity in 2010, are small and can be cached for a long period without cache invalidations. A query result cache is of huge benefit and memcached is an excellent solution.
 
Recent Activity changes frequently so a query cache is frequently invalidated, but regenerating the summary of Recent Activity is quite fast. Here a row cache helps further boost query performance. We rely on the InnoDB buffer pool in RAM and our own Flashcache kernel driver to expand the OS cache onto a flash device.
 
Developing in parallel
Timeline started as a Hackathon project in late 2010 with two full-time engineers, an engineering intern, and a designer building a working demo in a single night. The full team ramped up in early 2011, and the development team was split into design, front-end engineering, infrastructure engineering, and data migrations. By doing staged and layered prototyping, we achieved an amazing amount of development parallelism and rarely was any part of the team blocked by another. Early on in the project we were simultaneously:
 
1. Designing UI prototypes with our pre-existing but non-scalable backend,
2. Building production frontend code on a simulation of the scalable backend,
3. Building the scalable backend using samples of de-normalized data from a prototype of denormalization migration,
4. Building the framework to run the full-scale denormalization process,
5. Collecting and copying the data necessary for the denormalization,
6. Performing simulated load testing to validate our capacity planning estimates.
 
In retrospect, that’s pretty crazy. We had to move a lot of mountains to go from the initial infrastructure review meeting to successfully turning on the 100% backend load test in just six months. Done another way, this project could have taken twice as long - and that’s being generous.
 
Timeline has been a wonderful opportunity to work closely with the product team. Our constant collaboration was critical to ensuring the infrastructure we built supported their product goals while simultaneously guiding the product toward features that we could implement efficiently.
 
As millions of users enable Timeline, it is wonderfully exciting to see all the positive feedback and even more exciting to see our performance graphs look just like our simulations.
 
Ryan Mack, an infrastructure engineer, looks forward to rediscovering this blog post on his timeline a decade from now.
Chrissy Blue
TERRIBLE!
Jay Zalowitz
Good read
Vibeke Jyndevad Hauge
very messy
Aina N Tariq
interesting.
Ukeme Utuk
i thought this timeline was already ready for fb users to start using
Victoria Heckler
Love it. Good work FB geniuses!
Mouâd Edde
OKaay OKaaay!!!
Jym McMurdo
Awesome job guys. Any chance at seeing a search interface for Timeline? Often, I don't remember when I posted something, and it becomes difficult to find.
Karman Chua
I've been doing the same thing since the very beginning...even before FB!!LOL
Peter Whitehouse
but for some reason I NO LONGER have timeline - no one is helping me find out why
Lee Walsh
Yes, but....I still hate it!! I have family and friends for memories and photo albums for that stuff. I don't want to know what my best status update of 2009 was, I simply don't care! I'm never going to say something that profound that I need to log it and save it for my grand children. Facebook, you have left yourself open for a new competitor by not allowing the choice of timeline or not. I've not spoken to one person I know who actually likes it!
Lugh Eile
It's like Myspace, rubbish!
Peter Whitehouse
...and now it mystically reappears, thenak you fb
Being Diwash
terrible
Nishant Shah
(y)
Amol Anmol
It's facebook's new face...!!
Steven Collins
give up on timeline, fb, no one wants or likes it. theres a reason we left myspace
Victoria Heckler
Lee Walsh - Your complaint makes no sense. Facebook already retained every single post you made pre-timeline ... you just had to scroll back far enough to see them. All timeline does in this regard is group those posts by year, so you can find them if you wish to do so.
Jean Campos Ribeiro
Muito bom!
Nahar Gostu Addimawi
i dont know
Lee Walsh
Yes, but I don't wish to do so. And the new timeline is basically a method for me to do that. I want to see whats happening now and what has been happening in the past few weeks, not years. If i wanted to do that, which I don't, I would go and search through the "older posts" section of the old facebook. So, my post does make sense. The layout is horrific too, very messy and actually less like a timeline than the previous version which was stacked post by post, none of this left, right, here, there crap! Facebook, DISLIKE!
Mike Armstrong
Logically it works... but there's plenty more efficient ways of doing it :L
Зухайр Бедлам
it's great work and now is better than was ago! (y)
Daniel Tangente
Good job, and that is really amazing indeed. The only thing I hated is that you do not give option to stick with the old version once you enable timeline!!!.
Aric Bolf
Timeline is awesome! Thank you :)
Victoria Heckler
Lee Walsh - No, your post still doesn't make sense. If you don't wish to see your old posts, then don't look for them. That simple. Perhaps you just need to sort your posts so you can see the more recent ones on top, rather than the highlighted stories?
Nicole Fougerousse- Turley
I love the new time line way to go :)
Daniel Gutierrez Gileta
I feel naked now :(
Daniel Gutierrez Gileta
MAKE IT OPTIONAL! D:
David Moors
An amazing piece of engineering. Congratulation to all those involved, even if some users don't like.
Daniel Gutierrez Gileta
@David Moors some? most users don't like it, believe me.
Shon Lacy Burns
It's growing on me. ( Now I have to make a cool cover.)
Tom Bielecki
You've got some great architecture ideas going on, and I'm impressed with how you're helping to improve database systems like MySQL.

Personally, my biggest pet peeve with the introduction of the Timeline design is the verbage used to describe it (obviously that's a different team than you are on). Many of us began to use Facebook as a tool for communication, but you (Facebook) have begun to explicitly describe it as a data store. And it feels violating to be told (nearly explicitly) as a user that "you've given us so much of your life data that we can programatically filter it into your Life Story™ [muahaha]".

It's brilliant to use redesigns to redirect negative attention away from the shifts of core values of the service, but it's an evil sort of brilliant social coercion towards eroded views of privacy that Edward Bernays would be rolling around in his grave in jealousy of. I certainly think it's morally wrong that you have the new ability to target consumers based on "death of loved ones" and "broken bones," but it's hard not to applaud the idea that you've created a system where people want to provide these things because it adds social value.
Paul Undernet
When we export our data, we only get a partial set of past wall posts. We either need the complete export back or a way to search our own posts. Searching would be preferable.
Mary Beers
Facebook is no longer a nice place to just have fun with your friends, it is a place that anything you say or do on here is broadcast to more than just your friends, no matter how you set your privacy settings. For all time...ugggggggggg. & I agree with Daniel^^MAKE IT OPTIONAL!!!
Jael A Ruiz L
I can improve this thing jajaja
Yiannakis Yianni
Well, most I know are happy with it. Me, I'm happy too but still needs some work on it. Don't like this full in your face boxes of what's posted. It's way too big and the uneven sizes make the page look messy. Can follow the way posts go but boxes need to be even to keep the flow. Still need to see what I could do to it but overall happy with it all. Also, improved security too. ;)
Joseph Goble
it's a series of tubes!
Steven Briggs
Honestly, that is some incredible engineering. While a bunch of people sit here griping about the interface, I'll just sit back and admire the brilliance of how this was accomplished. Nice work.
Ryan Andersen
@Jym Mcmurdo Ctrl f would help you out there. And i agree. This is quite astonishing. Too many people expect the UI to be perfect.
Diegoo González
Basura! :/
David Tenser
From an engineering point of view, I can definitely see how this must have been a major undertaking -- and all I can say is that I'm impressed with the overall work making it possible! But from a user point of view... I guess I don't really see the point. Sure, it's nice to be able to quickly go back and see old posts if you want to get sentimental, but that could have been implemented in a couple of days by just allowing you to jump to a specific point in time in your profile -- no aggregation necessary. So it just seems like a ton of engineering work for little extra benefit. But now that it's already done, congrats! And please fix the sluggish scrolling now, ok? ;)
Fuji Brenda O'neal Fougerousse
I hate, hate, hate it.plus everything u do or say people that's not even your friend can see it. there is no privacy no matter how u set your privacy settings.
Lori Eby
Mine is obviously screwed up. I let the page fully load, then scroll down only to have it constantly pop it's way right back to the top. I got it over a week ago..can never scroll to read anything..just keeps bouncing it's way to the top again. ugh.
Lori Eby
Guess I won't have to worry about crap I posted a year ago...can't keep the page scrolled down long enough to find anything.
James Churchman
Firstly, Timeline is amazing... the best change to Facebook, after adding chat, so far .. elegant simple concise and beautiful i love it ..... but msql, in 2012? oh dear...
Josh Lacey
How do I get it ?
Ludwig Behm
the frontend (js/html) needs many corrections... where I could report bugs?
is there a blog or something where I can see the smallest updates at the design, too? I've written a dark colored style for the Facebook layout, but you change so many things... about in every hour... if there would be a protocol or changelog I could see witch part of the site I must check for complications with my design...
for interests: http://userstyles.org/styles/57096/facebook-dark-design-beta
Hal Cooper
Lee Walsh, don't scroll back if you don't want to. You said you could use "older posts" to go back, well, you'd be scrolling forever. To all you complaining, this will be like every other facebook release, a majority will complain, and then a majority of them will grow tired of complaining, and soon not be able to without it. Timeline just makes your older data easier to get to, simple as that, didn't hurt anything else. now, search would be nice, but that's a far bigger undertaking than this project, they used to have it, but too much data to search through, I was pissed the day that went away ;-)
Yasin Ibrahim
timeline is rubbish and terrible, you don't know what is going on on your profile untill you try your old facebook. lets demonstrate against time line.
Fuji Brenda O'neal Fougerousse
mine too lori eby
Lord Mada
انت كويس
Shęėñ Śm
is there an option of removing TIMELINE if we don;t want it on our page. please advice asap... its really annoying me
Nick Marsh
My biggest problem with Timeline is simply how laggy the thing gets as it's pulling data. Makes me want to not look back and find info.
Hassan Hamade
GOOD WORK
Kenji Tanaka
Wonderful! good job!
Vivian Gialanella-Sauter
Well, maybe you can re-think some of your engineering. You guys spent alot of time and put alot of work into this - why not have a great result? Timeline is confusing and cluttered. It needs to be SIMPLE, EASY TO READ, and PLEASANT TO THE EYE. If you will not consider making it optional - then, please improve it. You have alot more work to do.
Eduardo Asta
Timeline seems fine but it lacks some design features and the data set could be more interesting...I mean, you are losing the opportunity to help people do what they really want: tell their stories...
Ameet Kadamba
TimeLine makes sense of time...well done!
Srinivas Kulkarni
Interesting post, on the UI front, it may be terrific.... but yea, as mentioned, there are some improvisations that can still happen
Joseph Miller
I love the way PHP array functions work! PHP makes big data fast and dynamic...
Ankur Oberoi
inspirational effort on this engineering problem!
Richard Corfield
Very interesting read. Sounds like you've gone a bit more data warehouse with the denormalised structure.
Maroško Juzik Jusko
suck
Duke Ekpo
though i m yet to use the feature ;but do appreciate its prowess and the egg heads behind it
Nikunj D Sakhrelia
so normalization not work on each scenario good to know...
Robert Knox
The timeline is neat, I'll give you guys that, but what facebook really needs to work on is how they deal with their reports. You got people on here constantly harassing others, sending them nude disgusting photos, stealing their pics, and its not just happened to me. All they have to do is verify their account with their phone number, and their back to making people not want to use facebook anymore.
Karman Chua
DO I make the first cut..If I can reverse TIME?!LOL
Leslie Morss Hoyt
Is anyone at the FACEBOOK OFFICE readign the comments on the FB communties? A lot of us HATE TIMELINE - It's an ugly page format and not user friendly - I've been trying to give it a chance but nothing works right -I appreciate the technology aspect but some of us just liked the nice simple format of FB and would like it back PLEASE!!!!! After all we the subscribers have made FB what is - Try being more in touch with the FB community...
Leslie Morss Hoyt
PS - One has to do waaay too much navigation on TIMELINE - CRAPPY APPY!!!
Mattias Fjalestad
It is OK if you don't understand this article but for us who does...great work!
James Henson
Another hater of Timeline in case my son is listening
Mukhamed MaMdhu
wt-f... feat of engineering it maybe, I lost a lot of important previous posts, when Timeline took over. Explain that wilya... Whatever are we fighting for if not Operational Integrity- that which is of value to users.
Emilio Cecconi
Excellent writing. Thank you.
Michael Hight
thank you for providing a diary of all my life! I am sure my kids will enjoy!
مصطفى عادل
Hard Job I know... But Users faces some problems like missing older posts which hits in the reliability of Timeline, and facing performance issue is a threshold point that you should ALWAYS review.
Recreation Point
like this page plz
Annie Flores
dont like my timeline please remove\
Eric Lagier
@Scott Hollomon - did you try Memolane to see and search you social media history?
Dosso Charly
c'est bien j'aime mais je ce pas il sa gi de quoi llllllllllllllllll dddddddddddddd ooooooooooooooo fffffffffffffffff
Keisuke Ishikra
I'll try it.
Abhijit Ghatnekar
Very nice read and I appreciate the tech team's efforts. Especially the denormalization, the extent to which caching is done, the row cache approach adopted for data that's constantly changing.. But you guys gotta work a bit more on privacy settings of the Timeline. See.Sometimes, people post their personal inhibitions, frustrations, personal problems as status updates and Timeline provides a chronological display of those. At some time in future, when all is well, I would not like people to "see" those, you can understand it's a kinda embarrassing....ain't it??
কল্যাণী রমা
This is simply amazing!!!
Gerry Corbin
HOW DO I GO BACK TO THE OLD FB ONCE I DONT LKE TIMELINE
Hakeem Wahab
Good work and good choice of MySQL :)
Chouaib Founani
GOod Job Marck
Manish Gattani
bakwasssss..........
Nitish Kumar
timeline as a notbook or presonal diary of all ........thats better and store of tours...
Venu Gopal P. Venkata
Excellent... good job my friend
James K Steele
Why won't Timeline let me add multiple "First Met" entries? Every time I try, the name I put in the "Who" box goes all wonky and it wont let me continue.
Takenoshin Yamamoto
MySQL is a good choice at all...
Ricky Poh
how to remove timeline if i accidentally click the link, because i prefer previous facebook display template?
Aya Knanis
i test timeline, i think ,it has some mistakes... hope it gonna be improved soon ...
Gerry Corbin
HOW DO WE GET RIDE OF TIMELINE, AND GO BACK TO THE ORIGINAL WALL.
Kimberly Ronquillo Mendoza
ilike it :)\
Salman Khan
how can i get this
Amanda Fleming Taylor
Cool! I like it.
Eng Frank Firstkenyan
I think I wil B one of th last FB users 2 adopt th timeline. As a web app UI designer n backend coder I think th two column layout sucks. Too much info hits th user at a glance. I however appreciate th work tht went into creating it. Wish people kkud hv th optiob of opting out.
Brian Handscomb
Moderately interesting article, and actually has confirmed some suspicions on how Timeline came about. So much effort though in something I personally think is pointless, confusing, cluttered, FAR too heavily dependent on Javascript for the UI (slowdowns all round) and has actually given me reason to switch to IE7 (or prior) where everyone's profile page is still a wall...
Roe Roe
well , I'd like to know why I cannot see any of my previous post anylonger, my profile page can be viewed by everyone but me//???? what happeneddddddddddddd??
Sue Bowie
Don't understand how to get around much yet. My son hated it-- I'm not far behind~~~
Makayla Rene Howard
To mAke it yo profile
Imran Shami
at present i am using timeline, it really good.But i want to restore my old face book page.Can anyone can help me in this?
Roe Roe
my timeline dissappeared...now what??? upset to say the least
Criss Moon
>
I want to remove my Timeline. How can i remove my Timeline.
Please Let me know :(
Kurt Linderoos
Great development! To win this G+ really has much to do now.
HiSham Mansour Sayed
if u colud to help me bec i cant do time line plzzzz help mee
Billy Pig
Good work and good choice of MySQL, one problem,... keeps reloading on scroll-down.
Gautham Srikanth
How to see my subscribers list on my timeline? I recently changed into timeline and I m unable to see it? where can I find it in my profile?
Lynda G. Johnson
Timeline doesn't feel like facebook....
I don't have it, I don't want ....
Why not just turn it into it's own stand alone site ?
It reminds me of a scrapbook put together by an artsy kinda of Mom.
Call it timeline book.....
Just an honest opinion....
Emrah Güven
timeline is a waste of time but the biggest chit is enforcing all users to migrate timeline thingie!! it could or should be optional
Daniel Dee Rich Richardson II
I cant get Timeline and I really would like to have it please!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Djj Okan
yarrak
Adriana Adriana
hello! Can anyone tell me how can i remove timeline?
Eleanor Poulter
People who are sequential, linear thinkers prefer the old FB format, in chronological order. That's me. Timeline appeals to people who are more "random global" thinkers - the kind who also like those messy "mind maps", which I find boggling.
By the way, I tried a cyber "archaeological dig", scrolling back to 2010 for something, and guess what? Only a few random posts are there, one or two per month, and not what I was looking for. All of 2011 was still there. So if you want to keep any of your posts on FB, copy and paste them into a doc before FB removes them.
Kenneth M. Ngoy Kalolo
Is it MySQL then? forgive my ignorance of data centers technologies but how does Hadoop fit in Timeline?
Marlin Ouverson
Dear Facebook: when I hide something or delete something from my Timeline (or Wall), I expect it to _stay_ hidden or deleted and not keep re-appearing! This recurring problem in your system needs to be fixed ASAP, there are serious ramifications.
Mike Henry
You make it sound like your building the Great Wall of China...all I know is you got too many staff and too much innovation.STOP IT!!!
Mary Kay Erwin
Where can I find the answer to Gerry Corbin's question "HOW DO WE GET RID OF TIMELINE AND GO BACK TO THE ORIGINAL?"
Thee Stoner
How did u get it on i want it
Francesca Francesca
off timeline please ...
Narendra Dhami
Great! but not working smoothly ... had few bugs and need to be fixed first.
Anna Savvides
Sounds a lot like "Inception" minus the plot and sub-text.
Raheel Khan
Time line Failed....
Cassandra Armand
How do I get the timeline
Rodick Andirodick
timeline please ..,....
Barbara Harling
why can't we have a option to have timeline or keep it like it is why do we have to have it? I like it like it is.
Barbara Harling
another thing u could have explain it so we could understand it. And why r people saying we have 7 days to make changes? And what changes am I suppose to be making? Just don't understand it!!!!!!
Mary Fowler
Please at the very least make it optional
THE FORMAT IS HORRIBLE! Eveything Facebook is not supposed to be - cluttered and confusing. Facebook's formula to success was clean - easy on the eyes and easy to use format. Another former social networking giant - MySpace- failed because of their hard to use cluttered format. Timeline is just that way - cluttered, messy - difficult to manuver and painful to look at. Not at all enjoyable - which is what the whole experience is supposed to be.
Elbaz El Baz
cool i lik it
Elbaz El Baz
verry good
Gary Chen
Great!
Ashish Khanal
How to get timeline?
Becky OtterBot
^^ search for in under "help". It will walk you thru the process
Joyce Camille Wann
I love the timeliness mobile
Nick Martin
Seems quite elegant, esp like the parallelism for the join queries and the aggregation generators.
Seraj Demha
Why only image is in attach option in landscape size allowed in new Timeline, isn't a good idea to attach there small video/clip .
Albert Agbulos
Love timeline but HATE how there is no central location where all my videos are located on the MOBILE application on Android (not sure if other OS's have the same problem). HATE having to search my timeline wall to find my videos. Please consider a separate tab or link to videos? Thanks!
Lucy Mummy Lawton
How do I get timeline on my blackberry?
Suzanne Lemire
cant get timeline.
Chanté Roux
How can I get a timeline on my blackberry? And how do I get a timeline?
Rose Bright Boyd
You may think this is a bold now engineering effort, but people shouldn't be forced to have it.
Jason Pemberton
I don't like you, much.
Tracy Ann
y cant we get a choice...thr r manyyy who hate timeline,myself included
Kyon Anagata
i can't get an invitaion in timeline on how to get it. plsssssssss..... help me
Prethi Viswanatan
Great job !!!!
Glen Flower
let us turn it off. Its crap
Stephen Hill-mends
Hwew
George Mejia
I feel comfortable, but I prefer to try something new for the other users !
Adem Aziz
ahla
Ritagalih DInda
:-)Yukon
Brian Handscomb
Timeline has always seemed a buggy system, and note the fact that to get it to work at all there is a distinct 'Timeline copy'... Well this just asking for trouble, and in fact I have someone on my friend list with a page with almost all the content supposedly gone, but with a certain browser extension I can see all the original data in the Wall view clear as anything! Buggy Buggy Buggy Unwanted Feature, with lots of problems THAT FACEBOOK ACKNOWLEDGE on the help pages and STILL they are continuing to roll this out to people who have not yet converted their personal profile over (e.g. me, forced conversion to the layout of evil in a little under a week).
Tatenda Gora
Timeline sucks BIG time!!!!!!
Erme Caraballo
No entiendo ooo
Dung Truong
Timeline ? Nice idea but it's not for human. The timeline indeed does not get people together or understand more each other, it automatically makes the owner just sexier and more modern, it's useless completely. Anyway, timeline is good for marketing, and i think it's main purpose of the timeline, attract more marketers to deliver good pictures to customers.
SI MO
need to talk to you
Nathan Xin
Hi, I thought Facebook uses Cassandra for all. It is a little surprise that MySQL/InnoDB is the choice. Could you please explain briefly why ? thanks
Ben Wong
wow!