Manual testings looks good, working on automation.

SUCCESS: http://jenkins.ceph.dachary.org/job/ceph/8438/

• SUCCESS http://jenkins.ceph.dachary.org/job/ceph/LABELS=centos-7&&x86_64/8438/
• SUCCESS http://jenkins.ceph.dachary.org/job/ceph/LABELS=ubuntu-14.04&&x86_64/8438/

Hi @athanatos , this is ready for review.

One thing that is missing is a test case for unfound objects in replicated pool, I didn't figure out a way to do it easily. (I tried to remove all three copies from file system and then did a repair of the PG, it did not show unfound object, I guess that is another bug).

FAILURE: http://jenkins.ceph.dachary.org/job/ceph/8480/

• FAILURE http://jenkins.ceph.dachary.org/job/ceph/LABELS=centos-7&&x86_64/8480/

FAILURE: http://jenkins.ceph.dachary.org/job/ceph/8482/

• FAILURE http://jenkins.ceph.dachary.org/job/ceph/LABELS=centos-7&&x86_64/8482/

@dachary , the bot failure seems unrelated to the changes in this PR, is it a known pattern?

@guangyy I saw this false negative once yesterday but could not figure it out. Could you please rebase and repush to verify it goes away ? Thanks for letting me know :-)

Thanks @dachary , rebased and repushed, let us see the check this time :)

SUCCESS: http://jenkins.ceph.dachary.org/job/ceph/8514/

• SUCCESS http://jenkins.ceph.dachary.org/job/ceph/LABELS=centos-7&&x86_64/8514/

Ping @athanatos :)

As discussed with @athanatos , we will add a test case into ceph-qa-suite for this.

Hi @athanatos ,
If the code looks good, could you help to push it to ceph's repository, so that I can run some teuthology test? Thanks!

@guangyy I've added you to the ceph project, so you can push wip- branches. Be careful not to touch the main ones!

Alternatively, you can use teuthology-openstack, which does its own local builds and doesn't rely on any upstream access.

@guangyy for your convenience, here is the command line you need to run the test:

teuthology-openstack --verbose --key-name myself --key-filename ~/Downloads/myself --suite rados/thrash --ceph-qa-suite-git-url https://github.com/ceph/ceph-qa-suite.git --suite-branch ceph --ceph-git-url https://github.com/guangyy/ceph.git --ceph wip-13441 --subset 2/3

to verify that teuthology-openstack works for you, please follow the instructions at https://github.com/dachary/teuthology/tree/openstack#openstack-backend

Let me know if you need help.

Thanks @liewegas @dachary , I will give it a try.

Newly added ceph-qa-suite passed, ready for review and merge.

Test case: ceph/ceph-qa-suite#755
Job:
http://qa-proxy.ceph.com/teuthology/yguang-2015-12-04_17:18:17-rados-wip-13441---basic-multi/1167683/teuthology.log

@athanatos .

lgtm, does it still need a full suite run?

Thanks @athanatos , yeah I think it would be helpful to run along with a full suite.