Ceph

Confirmed on master as well.

The cache agent does not wake up when a proxy read happens and promotes the object in ReplicatedPG::maybe_handle_cache. The last manifestation of the agent is when OSDService::agent_entry found an empty queue and started waiting on agent_cond.

2015-08-12 18:03:26.195127 7f2e7f7ae700 20 osd.2 17 agent_entry empty queue

2015-08-12 18:03:52.175635 7f2e959ab700 10 osd.2 pg_epoch: 17 pg[2.0( v 17'5 (0'0,17'5] local-les=11 n=1 ec=10 les/c 11/11 10/10/10) [2,1,0] r=0 lpr=10 luod=17'4 lua=17'4 crt=16'3 lcod 16'3 mlcod 16'3 active+clean] agent_choose_mode flush_mode: idle evict_mode: idle num_objects: 1 num_bytes: 83 num_objects_dirty: 0 num_objects_omap: 0 num_dirty: 0 num_user_objects: 0 num_user_bytes: 0 pool.info.target_max_bytes: 0 pool.info.target_max_objects: 1
2015-08-12 18:03:52.175641 7f2e959ab700 20 osd.2 pg_epoch: 17 pg[2.0( v 17'5 (0'0,17'5] local-les=11 n=1 ec=10 les/c 11/11 10/10/10) [2,1,0] r=0 lpr=10 luod=17'4 lua=17'4 crt=16'3 lcod 16'3 mlcod 16'3 active+clean] agent_choose_mode dirty 0 full 0

The hit_set_period value of 5 is super low.. you probably want 600 at a minimum (10 minutes). But I'm not sure that's your problem.

Another possibility is that there are so few user objects in the pool vs the hit_set objects (1 + 1?), and there's an off-by-one or rounding error in the code that prevents the agent from busy looping trying to flush/evict. Or that a histogram of 1 object isn't behaving (e.g., showing that 1 object as 100th instead of 0th percentile vs the configured threshold).

Try putting more objects in the pool and see if the effect is still there? And increase the hit_set_period?

Loïc, just want to rectify some points here.

agent_choose_mode flush_mode: idle evict_mode: idle num_objects: 1 num_bytes: 83 num_objects_dirty: 0 num_objects_omap: 0 num_dirty: 0 num_user_objects: 0 num_user_bytes: 0 pool.info.target_max_bytes: 0 pool.info.target_max_objects: 1

so there was 1 hit_set_archive object which took 83 bytes. and this sort of objects is not taken into the consideration of cache eviction. as what we care about is only the number/size of user objects. that's the root cause why we failed to evict this non-user object in your test.

The cache eviction mode determined by ReplicatedPG::agent_choose_mode does not count the objects, it relies on ratios.

yes, so we need to take following things into consideration

• the target_max_bytes/target_max_objects are per-pool settings, but we are doing the evict at the PG level. so we normalize the number using pg_num. but your pool's pg_num is 1. so we are fine.
• even if the number of user object is just 1, we can still have it evicted. as long as it is a genuine object.

probably we should update the section of "ceph df" section in manpage of "ceph" to clarify that there could be some objects are non-user object, for example, are used by hit_set archiving and/or objects-omap.

Running the script in the description, here is what I find in the logs: when the agent_choose_mode function runs at 2015-08-13 11:47:33.129187 it finds there are num_user_objects: 0. Later on at 2015-08-13 11:47:33.131668 finish_promote 2/6cf8deff/obj1/head happens and if agent_choose_mode was to be run again, it would find that there is one user_objects. But it does not run and does not get a chance to reconsider.