7. Some optimization
ārulesā
- Donāt rely on being able to tweak anything.
8. Some optimization
ārulesā
- Donāt rely on being able to tweak anything.
- Donāt waste too much time tuning when
you have no evidence itāll matter.
10. Optimization ārulesā
āWe should forget about small efļ¬ciencies,
say about 97% of the time: premature
optimization is the root of all evil.ā
Knuth, (or Hoare)
15. Optimization ārulesā
That doesnāt give us an excuse to be lazy and
inefļ¬cient.
We lean on the experience of people in the
community for evidence that tuning(s) might
be a worthwhile thing to do.
20. Our Context
- 24 TB of MySQL data
- 32k/sec of MySQL writes
21. Our Context
- 24 TB of MySQL data
- 32k/sec of MySQL writes
- 120k/sec of MySQL reads
22. Our Context
- 24 TB of MySQL data
- 32k/sec of MySQL writes
- 120k/sec of MySQL reads
- 6 PB of photos
23. Our Context
- 24 TB of MySQL data
- 32k/sec of MySQL writes
- 120k/sec of MySQL reads
- 6 PB of photos
- 10TB storage eaten per day
24. Our Context
- 24 TB of MySQL data
- 32k/sec of MySQL writes
- 120k/sec of MySQL reads
- 6 PB of photos
- 10TB storage eaten per day
- 15,362 service monitors (alerts)
26. Infrastructure Hacks
- Examples of what changing software can do
(plain old-fashioned performance tuning)
27. Infrastructure Hacks
- Examples of what changing software can do
(plain old-fashioned performance tuning)
- Examples of what changing hardware can do
(yay for Mr. Moore!)
28. Leaning on compilers
(synthetic PHP benchmarks, not real-world)
(http://sebastian-bergmann.de/archives/634-PHP-GCC-
ICC-Benchmark.html)
32. Image Processing
- 2004, Flickr was using ImageMagick for
image processing (version 6.1.9)
33. Image Processing
- 2004, Flickr was using ImageMagick for
image processing (version 6.1.9)
- Changed to GraphicsMagick, about 15%
faster at the time (version 1.1.5)
34. Image Processing
- 2004, Flickr was using ImageMagick for
image processing (version 6.1.9)
- Changed to GraphicsMagick, about 15%
faster at the time (version 1.1.5)
- Only need a subset of ImageMagick features
anyway for our purposes
35. Image Processing
- OpenMP support
(http://en.wikipedia.org/wiki/Openmp)
- Allows parallelization of processing jobs,
using multiple cores working on the same
image
- Some algorithms have more parallelization
than others
36. Image Processing
- Test script
- 7 large-ish DSLR photos
- Cascade resizing each to 6 smaller sizes,
semi-typical for Flickrās workload
- Each resize processed serially
37. Image Processing
compiler differences
(GM version 1.1.14, non-OpenMP)
45. Diagonal Scaling
67 āoldā webservers with 18 ānewā :
We replaced
46. Diagonal Scaling
67 āoldā webservers with 18 ānewā :
We replaced
CPUs RAM drives total power (W)
servers @60% peak
per server per server per server
67 2 8763.6
4GB 1x80GB
18 8 2332.8
4GB 1x146GB
47. Diagonal Scaling
67 āoldā webservers with 18 ānewā :
We replaced
CPUs RAM drives total power (W)
servers @60% peak
~70% LESS power
per server per server per server
67 2 8763.6
4GB 1x80GB
49U LESS rack space
18 8 2332.8
4GB 1x146GB
49. Diagonal Scaling
23 āoldā image processing boxes with 8 ānewā
We replaced
50. Diagonal Scaling
23 āoldā image processing boxes with 8 ānewā
We replaced
server photos/min rack total power (W)
@60% peak
23 1035 23 3008.4
8 1120 8 1036.8
51. Diagonal Scaling
23 āoldā image processing boxes with 8 ānewā
We replaced
server photos/min rack total power (W)
@60% peak
~75% FASTER
23 1035 23 3008.4
15U LESS rack space
65% LESS power 8
8 1120 1036.8
52. Diagonal Scaling
23 āoldā image processing boxes with 8 ānewā
We replaced
server photos/min rack total power (W)
~75% FASTER @60% peak
15U LESS rack space
23 1035 23 3008.4
65% LESS power 8
8 1120 1036.8
53. Diagonal Scaling
23 āoldā image processing boxes with 8 ānewā
We replaced
server photos/min rack total power (W)
~75% FASTER @60% peak
15U LESS rack space
23 1035 23 3008.4
65% LESS power 8
8 1120 1036.8
54. Diagonal Scaling
23 āoldā image processing boxes with 8 ānewā
We replaced
server photos/min rack total power (W)
~75% FASTER @60% peak
15U LESS rack space
23 1035 23 3008.4
65% LESS power 8
8 1120 1036.8
from this
to this
60. Ofļ¬ine Tasks
- Out-of-band/asynchronous queuing and execution
system, for non-realtime tasks
61. Ofļ¬ine Tasks
- Out-of-band/asynchronous queuing and execution
system, for non-realtime tasks
- See here:
62. Ofļ¬ine Tasks
- Out-of-band/asynchronous queuing and execution
system, for non-realtime tasks
- See here:
http://code.ļ¬ickr.com/blog/2008/09/26/ļ¬ickr-engineers-do-it-ofļ¬ine/
63. Ofļ¬ine Tasks
- Out-of-band/asynchronous queuing and execution
system, for non-realtime tasks
- See here:
http://code.ļ¬ickr.com/blog/2008/09/26/ļ¬ickr-engineers-do-it-ofļ¬ine/
- See Myles Grant talk about it more here:
64. Ofļ¬ine Tasks
- Out-of-band/asynchronous queuing and execution
system, for non-realtime tasks
- See here:
http://code.ļ¬ickr.com/blog/2008/09/26/ļ¬ickr-engineers-do-it-ofļ¬ine/
- See Myles Grant talk about it more here:
http://en.oreilly.com/velocity2009/public/schedule/detail/7552
69. Why?
As infrastructure grows, try to keep the
Humans:Machines ratio from getting out of
hand
Some of the How:
- teach machines to build themselves
70. Why?
As infrastructure grows, try to keep the
Humans:Machines ratio from getting out of
hand
Some of the How:
- teach machines to build themselves
- teach machines to watch themselves
71. Why?
As infrastructure grows, try to keep the
Humans:Machines ratio from getting out of
hand
Some of the How:
- teach machines to build themselves
- teach machines to watch themselves
- teach machines to ļ¬x themselves
72. Why?
As infrastructure grows, try to keep the
Humans:Machines ratio from getting out of
hand
Some of the How:
- teach machines to build themselves
- teach machines to watch themselves
- teach machines to ļ¬x themselves
- reduce MTTR by streamlining
74. Automated
Infrastructure
- If there is only one thing you do, automatic
conļ¬guration and deployment management
should be it.
75. Automated
Infrastructure
- If there is only one thing you do, automatic
conļ¬guration and deployment management
should be it.
- See:
- Opscode/Chef (http://opscode.com/)
- Puppet (http://reductivelabs.com/products/puppet/)
- System Imager/Conļ¬gurator
(http://wiki.systemimager.org)
77. Time
Machine time is cheaper than human time.
If a failure results in some commands being
run to āļ¬xā it, make the machines do it.
(i.e., donāt wake people up for stupid things!)
79. Aggregate Monitoring
Donāt care about single nodes, only care about delta
change of metrics/faults
- Warn (email) on X % change
- Page (wake up) on Y % change
80. Aggregate Monitoring
Donāt care about single nodes, only care about delta
change of metrics/faults
- Warn (email) on X % change
- Page (wake up) on Y % change
High and low water marks for some metrics
84. Self-Healing
Make service monitoring ļ¬x common failure
scenarios, notify us later about it.
Daemons/processes run on machines, will take
corrective action under certain conditions, and
report back with what they did.
85. Self-Healing
Make service monitoring ļ¬x common failure
scenarios, notify us later about it.
Daemons/processes run on machines, will take
corrective action under certain conditions, and
report back with what they did.
86. Self-Healing
Make service monitoring ļ¬x common failure
scenarios, notify us later about it.
Daemons/processes run on machines, will take
corrective action under certain conditions, and
report back with what they did.
Can greatly reduce your mean time to recovery
(MTTR)
87. Self-Healing
Make service monitoring ļ¬x common failure
scenarios, notify us later about it.
Daemons/processes run on machines, will take
corrective action under certain conditions, and
report back with what they did.
Can greatly reduce your mean time to recovery
(MTTR)
90. Basic Apache Example
1. Webserver not running?
2. Under certain conditions, try to start it, and
email that this happened. (Iāll read it
tomorrow)
91. Basic Apache Example
1. Webserver not running?
2. Under certain conditions, try to start it, and
email that this happened. (Iāll read it
tomorrow)
3. Wonāt start? Assume somethingās really
wrong, so donāt keep trying (email that, too)
95. MySQL Self-Healing
Some MySQL Issues āļ¬xedā by the machines
- Kill long-running SELECT queries (marked safe to kill)
96. MySQL Self-Healing
Some MySQL Issues āļ¬xedā by the machines
- Kill long-running SELECT queries (marked safe to kill)
- Queries not safe to kill are marked by the application
as āNO KILLā in comments
97. MySQL Self-Healing
Some MySQL Issues āļ¬xedā by the machines
- Kill long-running SELECT queries (marked safe to kill)
- Queries not safe to kill are marked by the application
as āNO KILLā in comments
- Run EXPLAIN on killed queries, and report the results
98. MySQL Self-Healing
Some MySQL Issues āļ¬xedā by the machines
- Kill long-running SELECT queries (marked safe to kill)
- Queries not safe to kill are marked by the application
as āNO KILLā in comments
- Run EXPLAIN on killed queries, and report the results
- Keep track of the query types and databases that need
the most killing, produce a āDBs that Suckā report
101. MySQL Self-Healing
Some MySQL Replication issues āļ¬xedā by the
machines, by error
- Skip errors that can safely be skipped and
restart slave threads
102. MySQL Self-Healing
Some MySQL Replication issues āļ¬xedā by the
machines, by error
- Skip errors that can safely be skipped and
restart slave threads
- Force refetch of replication binlogs on:
- 1064 (ER_PARSE_ERROR)
103. MySQL Self-Healing
Some MySQL Replication issues āļ¬xedā by the
machines, by error
- Skip errors that can safely be skipped and
restart slave threads
- Force refetch of replication binlogs on:
- 1064 (ER_PARSE_ERROR)
- Re-run queries on:
- 1205 (ER_LOCK_WAIT_TIMEOUT)
- 1213 (ER_LOCK_DEADLOCK)
109. Communications
ā¢ Internal IRC
- For ongoing discussions
- Logged, so āinļ¬niteā scrollback
110. Communications
ā¢ Internal IRC
- For ongoing discussions
- Logged, so āinļ¬niteā scrollback
ā¢ IM Bot (built on libyahoo2.sf.net)
- For production changes
- Broadcasts all to all contacts
- Logged, and injected into IRC
- IM Status = who is in primary/secondary on-call
111. Communications
ā¢ Internal IRC
- For ongoing discussions
- Logged, so āinļ¬niteā scrollback
ā¢ IM Bot (built on libyahoo2.sf.net)
- For production changes
- Broadcasts all to all contacts
- Logged, and injected into IRC
- IM Status = who is in primary/secondary on-call
ā¢ All of IRC and IM Bot slurped into a search index
126. Morals of Our Stories
- Optimizations can be a Very Good Thingā¢
127. Morals of Our Stories
- Optimizations can be a Very Good Thingā¢
- Weigh time spent optimizing against
expected gains
128. Morals of Our Stories
- Optimizations can be a Very Good Thingā¢
- Weigh time spent optimizing against
expected gains
- Lean on others for how much āexpected
gainsā mean for different scenarios
129. Morals of Our Stories
- Optimizations can be a Very Good Thingā¢
- Weigh time spent optimizing against
expected gains
- Lean on others for how much āexpected
gainsā mean for different scenarios
- Plain old-fashioned intuition
130. Some Wisdom Nuggets
Jon Prallās 85 WebOps Rules:
http://jprall.vox.com/library/post/85-
operations-rules-to-live-by.html
