Project

General

Profile

Actions
Copy link

Bug #14094

closed

ceph-disk list: OSError: [Errno 2] No such file or directory: '/dev/vdb1'

Added by Loïc Dachary over 8 years ago. Updated over 8 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
Loïc Dachary
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

After a failed run of the ceph-disk suite on CentOS 7.1, ceph-disk list tries to stat /dev/vdb1 although it does not exist. If run again immediately afterwards, the problem goes away. It is not a timing problem, nothing has been going one for over 15 minutes after the test stopped.

It showed twice but I'm not sure how to reproduce it reliably.

[root@target167114225077 ceph-disk]# ceph-disk list
Traceback (most recent call last):
  File "/sbin/ceph-disk", line 4054, in <module>
    main(sys.argv[1:])
  File "/sbin/ceph-disk", line 4010, in main
    main_catch(args.func, args)
  File "/sbin/ceph-disk", line 4032, in main_catch
    func(args)
  File "/sbin/ceph-disk", line 3324, in main_list
    devices = list_devices(args.path)
  File "/sbin/ceph-disk", line 3274, in list_devices
    ptype = get_partition_type(dev)
  File "/sbin/ceph-disk", line 3077, in get_partition_type
    return get_sgdisk_partition_info(part, 'Partition GUID code: (\S+)')
  File "/sbin/ceph-disk", line 3083, in get_sgdisk_partition_info
    (base, partnum) = split_dev_base_partnum(dev)
  File "/sbin/ceph-disk", line 3067, in split_dev_base_partnum
    if is_mpath(dev):
  File "/sbin/ceph-disk", line 460, in is_mpath
    uuid = get_dm_uuid(dev)
  File "/sbin/ceph-disk", line 448, in get_dm_uuid
    uuid_path = os.path.join(block_path(dev), 'dm', 'uuid')
  File "/sbin/ceph-disk", line 443, in block_path
    rdev = os.stat(path).st_rdev
OSError: [Errno 2] No such file or directory: '/dev/vdb1'
[root@target167114225077 ceph-disk]# ceph-disk list
/dev/vda :
 /dev/vda1 other, xfs, mounted on /
/dev/vdb other, unknown
/dev/vdc :
 /dev/vdc1 ceph journal
/dev/vdd other, unknown

Related issues 1 (0 open — 1 closed)

Related to Ceph - Bug #14080: ceph-disk: use blkid instead of sgdisk -iResolvedLoïc Dachary12/14/2015

Actions
Actions
Copy link
 #1

Updated by Loïc Dachary over 8 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Loïc Dachary over 8 years ago 

            part_uuid = get_partition_uuid(dev)
            if part_uuid:
                uuid_map[part_uuid] = dev
            ptype = get_partition_type(dev)

get_partition_uuid(dev) succeeds, meaning /dev/vdb1 exists and right after sgdisk -i 1 /dev/vdb it not longer exists. That suggests sgdisk -i 1 /dev/vdb has a side effect that immediately impacts the next call.
Actions
Copy link
 #3

Updated by Loïc Dachary over 8 years ago

Or just that a partprobe is ongoing and /dev/sdb1 is re-created while ceph-disk list. Maybe protecting ceph-disk list with a udevadm settle is all there is to do.

Actions
Copy link
 #4

Updated by Loïc Dachary over 8 years ago 

2015-12-17 11:54:49,077.077 INFO:tasks.workunit.client.0.target167114233028.stderr:        set -xe
2015-12-17 11:54:49,077.077 INFO:tasks.workunit.client.0.target167114233028.stderr:        ceph-disk --verbose deactivate --deactivate-by-id 3
2015-12-17 11:54:49,077.077 INFO:tasks.workunit.client.0.target167114233028.stderr:        ceph-disk --verbose destroy --destroy-by-id 3 --zap
2015-12-17 11:54:49,077.077 INFO:tasks.workunit.client.0.target167114233028.stderr:
2015-12-17 11:54:49,081.081 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:+ ceph-disk --verbose deactivate --deactivate-by-id 3
2015-12-17 11:54:49,174.174 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/udevadm settle --timeout=600
2015-12-17 11:54:49,183.183 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:list_all_partitions: vda
2015-12-17 11:54:49,183.183 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda uuid path is /sys/dev/block/253:0/dm/uuid
2015-12-17 11:54:49,183.183 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:list_all_partitions: vdb
2015-12-17 11:54:49,183.183 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdb uuid path is /sys/dev/block/253:16/dm/uuid
2015-12-17 11:54:49,183.183 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:list_all_partitions: vdc
2015-12-17 11:54:49,184.184 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc uuid path is /sys/dev/block/253:32/dm/uuid
2015-12-17 11:54:49,184.184 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:list_all_partitions: vdd
2015-12-17 11:54:49,184.184 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd uuid path is /sys/dev/block/253:48/dm/uuid
2015-12-17 11:54:49,184.184 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda1 uuid path is /sys/dev/block/253:1/dm/uuid
2015-12-17 11:54:49,184.184 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vda
2015-12-17 11:54:49,189.189 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda1 uuid path is /sys/dev/block/253:1/dm/uuid
2015-12-17 11:54:49,190.190 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vda
2015-12-17 11:54:49,201.201 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:main_list: /dev/vda1 ptype = 0fc63daf-8483-4772-8e79-3d69d8477de4 uuid = cfebdfb5-563f-4823-b419-a697480fc98d
2015-12-17 11:54:49,202.202 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc1 uuid path is /sys/dev/block/253:33/dm/uuid
2015-12-17 11:54:49,202.202 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdc
2015-12-17 11:54:49,211.211 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc1 uuid path is /sys/dev/block/253:33/dm/uuid
2015-12-17 11:54:49,211.211 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdc
2015-12-17 11:54:49,224.224 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:main_list: /dev/vdc1 ptype = 4fbd7e29-9d25-41b8-afd0-062c0ceff05d uuid = 9ad0073e-a4b4-11e5-b02e-fa163e8241b5
2015-12-17 11:54:49,224.224 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/blkid -s TYPE /dev/vdc1
2015-12-17 11:54:49,227.227 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:Mounting /dev/vdc1 on /var/lib/ceph/tmp/mnt.aY5whQ with options
2015-12-17 11:54:49,228.228 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/mount -t xfs -o  -- /dev/vdc1 /var/lib/ceph/tmp/mnt.aY5whQ
2015-12-17 11:54:49,236.236 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/restorecon /var/lib/ceph/tmp/mnt.aY5whQ
2015-12-17 11:54:49,241.241 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt.aY5whQ
2015-12-17 11:54:49,241.241 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.aY5whQ
2015-12-17 11:54:49,248.248 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd1 uuid path is /sys/dev/block/253:49/dm/uuid
2015-12-17 11:54:49,249.249 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdd
2015-12-17 11:54:49,262.262 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd1 uuid path is /sys/dev/block/253:49/dm/uuid
2015-12-17 11:54:49,263.263 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdd
2015-12-17 11:54:49,273.273 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:main_list: /dev/vdd1 ptype = 45b0969e-9b03-4f30-b4c6-b4b80ceff106 uuid = 8c61499d-457f-4a29-ae1c-be361940bfcf
2015-12-17 11:54:49,274.274 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd2 uuid path is /sys/dev/block/253:50/dm/uuid
2015-12-17 11:54:49,274.274 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 2 /dev/vdd
2015-12-17 11:54:49,286.286 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd2 uuid path is /sys/dev/block/253:50/dm/uuid
2015-12-17 11:54:49,286.286 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 2 /dev/vdd
2015-12-17 11:54:49,299.299 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:main_list: /dev/vdd2 ptype = 45b0969e-9b03-4f30-b4c6-b4b80ceff106 uuid = 08d91923-0d1a-4be2-a715-cea4c9068d3f
2015-12-17 11:54:49,299.299 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:main_list: {'vda': ['vda1'], 'vdc': ['vdc1'], 'vdb': [], 'vdd': ['vdd1', 'vdd2']}, uuid_map = {'cfebdfb5-563f-4823-b419-a697480fc98d': '/dev/vda1', '08d91923-0d1a-4be2-a715-cea4c9068d3f': '/dev/vdd2', '8c61499d-457f-4a29-ae1c-be361940bfcf': '/dev/vdd1', '9ad0073e-a4b4-11e5-b02e-fa163e8241b5': '/dev/vdc1'}, journal_map = {'08d91923-0d1a-4be2-a715-cea4c9068d3f': '/dev/vdc1'}
2015-12-17 11:54:49,300.300 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda1 uuid path is /sys/dev/block/253:1/dm/uuid
2015-12-17 11:54:49,300.300 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda1 uuid path is /sys/dev/block/253:1/dm/uuid
2015-12-17 11:54:49,300.300 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vda
2015-12-17 11:54:49,312.312 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda1 uuid path is /sys/dev/block/253:1/dm/uuid
2015-12-17 11:54:49,312.312 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vda
2015-12-17 11:54:49,316.316 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:list_dev(dev = /dev/vda1, ptype = 0fc63daf-8483-4772-8e79-3d69d8477de4)
2015-12-17 11:54:49,316.316 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/blkid -s TYPE /dev/vda1
2015-12-17 11:54:49,321.321 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdb uuid path is /sys/dev/block/253:16/dm/uuid
2015-12-17 11:54:49,321.321 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:list_dev(dev = /dev/vdb, ptype = unknown)
2015-12-17 11:54:49,321.321 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/blkid -s TYPE /dev/vdb
2015-12-17 11:54:49,679.679 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc1 uuid path is /sys/dev/block/253:33/dm/uuid
2015-12-17 11:54:49,679.679 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc1 uuid path is /sys/dev/block/253:33/dm/uuid
2015-12-17 11:54:49,680.680 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdc
2015-12-17 11:54:49,689.689 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc1 uuid path is /sys/dev/block/253:33/dm/uuid
2015-12-17 11:54:49,692.692 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdc
2015-12-17 11:54:49,698.698 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:list_dev(dev = /dev/vdc1, ptype = 4fbd7e29-9d25-41b8-afd0-062c0ceff05d)
2015-12-17 11:54:49,700.700 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/blkid -s TYPE /dev/vdc1
2015-12-17 11:54:49,706.706 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
2015-12-17 11:54:49,727.727 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd1 uuid path is /sys/dev/block/253:49/dm/uuid
2015-12-17 11:54:49,727.727 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd1 uuid path is /sys/dev/block/253:49/dm/uuid
2015-12-17 11:54:49,727.727 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdd
2015-12-17 11:54:49,738.738 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd1 uuid path is /sys/dev/block/253:49/dm/uuid
2015-12-17 11:54:49,739.739 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdd
2015-12-17 11:54:49,746.746 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:list_dev(dev = /dev/vdd1, ptype = 45b0969e-9b03-4f30-b4c6-b4b80ceff106)
2015-12-17 11:54:49,746.746 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd2 uuid path is /sys/dev/block/253:50/dm/uuid
2015-12-17 11:54:49,747.747 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd2 uuid path is /sys/dev/block/253:50/dm/uuid
2015-12-17 11:54:49,747.747 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 2 /dev/vdd
2015-12-17 11:54:49,754.754 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd2 uuid path is /sys/dev/block/253:50/dm/uuid
2015-12-17 11:54:49,755.755 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 2 /dev/vdd
2015-12-17 11:54:49,767.767 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:list_dev(dev = /dev/vdd2, ptype = 45b0969e-9b03-4f30-b4c6-b4b80ceff106)
2015-12-17 11:54:49,768.768 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:list_devices: [{'path': '/dev/vda', 'partitions': [{'dmcrypt': {}, 'uuid': '09113c14-0fd3-45d5-bd25-99b5305b1063', 'mount': '/', 'ptype': '0fc63daf-8483-4772-8e79-3d69d8477de4', 'is_partition': True, 'fs_type': 'xfs', 'path': '/dev/vda1', 'type': 'other'}]}, {'path': '/dev/vdb', 'type': 'other', 'dmcrypt': {}, 'ptype': 'unknown', 'is_partition': False}, {'path': '/dev/vdc', 'partitions': [{'dmcrypt': {}, 'uuid': '9ad0073e-a4b4-11e5-b02e-fa163e8241b5', 'mount': '/var/lib/ceph/osd/ceph-3', 'ptype': '4fbd7e29-9d25-41b8-afd0-062c0ceff05d', 'is_partition': True, 'cluster': 'ceph', 'state': 'active', 'fs_type': 'xfs', 'ceph_fsid': '7ca6c79d-e401-4b68-ad0f-f3b44f181b1e', 'path': '/dev/vdc1', 'type': 'data', 'whoami': '3', 'journal_dev': '/dev/vdd2', 'journal_uuid': '08d91923-0d1a-4be2-a715-cea4c9068d3f'}]}, {'path': '/dev/vdd', 'partitions': [{'dmcrypt': {}, 'uuid': '8c61499d-457f-4a29-ae1c-be361940bfcf', 'ptype': '45b0969e-9b03-4f30-b4c6-b4b80ceff106', 'is_partition': True, 'path': '/dev/vdd1', 'type': 'journal'}, {'dmcrypt': {}, 'uuid': '08d91923-0d1a-4be2-a715-cea4c9068d3f', 'ptype': '45b0969e-9b03-4f30-b4c6-b4b80ceff106', 'is_partition': True, 'journal_for': '/dev/vdc1', 'path': '/dev/vdd2', 'type': 'journal'}]}]
2015-12-17 11:54:49,768.768 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Checking osd id: 3 ...
2015-12-17 11:54:49,768.768 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/ceph osd dump --cluster=ceph --format json
2015-12-17 11:54:50,070.070 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:Stoping ceph osd.3...
2015-12-17 11:54:50,071.071 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/systemctl disable ceph-osd@3
2015-12-17 11:54:50,083.083 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:Removed symlink /etc/systemd/system/ceph-osd.target.wants/ceph-osd@3.service.
2015-12-17 11:54:50,169.169 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/systemctl stop ceph-osd@3
2015-12-17 11:54:53,238.238 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Remove `ready` file.
2015-12-17 11:54:53,238.238 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Remove `active` file.
2015-12-17 11:54:53,238.238 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup init
2015-12-17 11:54:53,250.250 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/ceph-detect-init --default sysvinit
2015-12-17 11:54:53,312.312 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Remove `systemd` file.
2015-12-17 11:54:53,312.312 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/restorecon -R /var/lib/ceph/osd/ceph-3/deactive
2015-12-17 11:54:53,324.324 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3/deactive
2015-12-17 11:54:53,327.327 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:Unmounting /var/lib/ceph/osd/ceph-3
2015-12-17 11:54:53,328.328 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/osd/ceph-3
2015-12-17 11:54:53,353.353 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Umount `/var/lib/ceph/osd/ceph-3` successfully.
2015-12-17 11:54:53,361.361 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:+ ceph-disk --verbose destroy --destroy-by-id 3 --zap
2015-12-17 11:54:53,443.443 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/udevadm settle --timeout=600
2015-12-17 11:55:00,547.547 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:list_all_partitions: vda
2015-12-17 11:55:00,547.547 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda uuid path is /sys/dev/block/253:0/dm/uuid
2015-12-17 11:55:00,550.550 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:list_all_partitions: vdb
2015-12-17 11:55:00,550.550 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdb uuid path is /sys/dev/block/253:16/dm/uuid
2015-12-17 11:55:00,550.550 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:list_all_partitions: vdc
2015-12-17 11:55:00,550.550 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc uuid path is /sys/dev/block/253:32/dm/uuid
2015-12-17 11:55:00,551.551 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:list_all_partitions: vdd
2015-12-17 11:55:00,551.551 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd uuid path is /sys/dev/block/253:48/dm/uuid
2015-12-17 11:55:00,551.551 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda1 uuid path is /sys/dev/block/253:1/dm/uuid
2015-12-17 11:55:00,551.551 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vda
2015-12-17 11:55:00,564.564 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda1 uuid path is /sys/dev/block/253:1/dm/uuid
2015-12-17 11:55:00,564.564 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vda
2015-12-17 11:55:00,572.572 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:main_list: /dev/vda1 ptype = 0fc63daf-8483-4772-8e79-3d69d8477de4 uuid = c61fa287-0238-4f44-a28c-3babec76072e
2015-12-17 11:55:00,573.573 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc1 uuid path is /sys/dev/block/253:33/dm/uuid
2015-12-17 11:55:00,573.573 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdc
2015-12-17 11:55:00,588.588 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc1 uuid path is /sys/dev/block/253:33/dm/uuid
2015-12-17 11:55:00,588.588 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdc
2015-12-17 11:55:00,958.958 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:main_list: /dev/vdc1 ptype = 4fbd7e29-9d25-41b8-afd0-062c0ceff05d uuid = 9ad0073e-a4b4-11e5-b02e-fa163e8241b5
2015-12-17 11:55:00,958.958 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/blkid -s TYPE /dev/vdc1
2015-12-17 11:55:04,832.832 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:Mounting /dev/vdc1 on /var/lib/ceph/tmp/mnt.4aAduf with options
2015-12-17 11:55:04,832.832 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/mount -t xfs -o  -- /dev/vdc1 /var/lib/ceph/tmp/mnt.4aAduf
2015-12-17 11:55:08,265.265 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/restorecon /var/lib/ceph/tmp/mnt.4aAduf
2015-12-17 11:55:08,333.333 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt.4aAduf
2015-12-17 11:55:08,334.334 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.4aAduf
2015-12-17 11:55:08,351.351 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd1 uuid path is /sys/dev/block/253:49/dm/uuid
2015-12-17 11:55:08,351.351 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdd
2015-12-17 11:55:08,376.376 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd1 uuid path is /sys/dev/block/253:49/dm/uuid
2015-12-17 11:55:08,376.376 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdd
2015-12-17 11:55:08,665.665 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:main_list: /dev/vdd1 ptype = 45b0969e-9b03-4f30-b4c6-b4b80ceff106 uuid = 8c61499d-457f-4a29-ae1c-be361940bfcf
2015-12-17 11:55:08,665.665 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd2 uuid path is /sys/dev/block/253:50/dm/uuid
2015-12-17 11:55:08,666.666 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 2 /dev/vdd
2015-12-17 11:55:08,674.674 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd2 uuid path is /sys/dev/block/253:50/dm/uuid
2015-12-17 11:55:08,676.676 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 2 /dev/vdd
2015-12-17 11:55:08,689.689 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:main_list: /dev/vdd2 ptype = 45b0969e-9b03-4f30-b4c6-b4b80ceff106 uuid = 08d91923-0d1a-4be2-a715-cea4c9068d3f
2015-12-17 11:55:08,689.689 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:main_list: {'vda': ['vda1'], 'vdc': ['vdc1'], 'vdb': [], 'vdd': ['vdd1', 'vdd2']}, uuid_map = {'08d91923-0d1a-4be2-a715-cea4c9068d3f': '/dev/vdd2', 'c61fa287-0238-4f44-a28c-3babec76072e': '/dev/vda1', '8c61499d-457f-4a29-ae1c-be361940bfcf': '/dev/vdd1', '9ad0073e-a4b4-11e5-b02e-fa163e8241b5': '/dev/vdc1'}, journal_map = {'08d91923-0d1a-4be2-a715-cea4c9068d3f': '/dev/vdc1'}
2015-12-17 11:55:08,689.689 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda1 uuid path is /sys/dev/block/253:1/dm/uuid
2015-12-17 11:55:08,689.689 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda1 uuid path is /sys/dev/block/253:1/dm/uuid
2015-12-17 11:55:08,689.689 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vda
2015-12-17 11:55:08,698.698 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda1 uuid path is /sys/dev/block/253:1/dm/uuid
2015-12-17 11:55:08,699.699 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vda
2015-12-17 11:55:08,710.710 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:list_dev(dev = /dev/vda1, ptype = 0fc63daf-8483-4772-8e79-3d69d8477de4)
2015-12-17 11:55:08,710.710 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/blkid -s TYPE /dev/vda1
2015-12-17 11:55:08,714.714 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdb uuid path is /sys/dev/block/253:16/dm/uuid
2015-12-17 11:55:08,714.714 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:list_dev(dev = /dev/vdb, ptype = unknown)
2015-12-17 11:55:08,715.715 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/blkid -s TYPE /dev/vdb
2015-12-17 11:55:09,936.936 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc1 uuid path is /sys/dev/block/253:33/dm/uuid
2015-12-17 11:55:09,937.937 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc1 uuid path is /sys/dev/block/253:33/dm/uuid
2015-12-17 11:55:09,937.937 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdc
2015-12-17 11:55:09,949.949 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc1 uuid path is /sys/dev/block/253:33/dm/uuid
2015-12-17 11:55:09,950.950 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdc
2015-12-17 11:55:09,963.963 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:list_dev(dev = /dev/vdc1, ptype = 4fbd7e29-9d25-41b8-afd0-062c0ceff05d)
2015-12-17 11:55:09,964.964 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/blkid -s TYPE /dev/vdc1
2015-12-17 11:55:12,833.833 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:Mounting /dev/vdc1 on /var/lib/ceph/tmp/mnt._3QP_m with options
2015-12-17 11:55:12,833.833 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/mount -t xfs -o  -- /dev/vdc1 /var/lib/ceph/tmp/mnt._3QP_m
2015-12-17 11:55:16,846.846 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/restorecon /var/lib/ceph/tmp/mnt._3QP_m
2015-12-17 11:55:16,864.864 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
2015-12-17 11:55:16,894.894 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt._3QP_m
2015-12-17 11:55:16,895.895 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt._3QP_m
2015-12-17 11:55:16,915.915 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd1 uuid path is /sys/dev/block/253:49/dm/uuid
2015-12-17 11:55:16,915.915 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd1 uuid path is /sys/dev/block/253:49/dm/uuid
2015-12-17 11:55:16,915.915 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdd
2015-12-17 11:55:16,941.941 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd1 uuid path is /sys/dev/block/253:49/dm/uuid
2015-12-17 11:55:16,941.941 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdd
2015-12-17 11:55:17,030.030 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:list_dev(dev = /dev/vdd1, ptype = 45b0969e-9b03-4f30-b4c6-b4b80ceff106)
2015-12-17 11:55:17,030.030 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd2 uuid path is /sys/dev/block/253:50/dm/uuid
2015-12-17 11:55:17,031.031 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd2 uuid path is /sys/dev/block/253:50/dm/uuid
2015-12-17 11:55:17,031.031 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 2 /dev/vdd
2015-12-17 11:55:17,039.039 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd2 uuid path is /sys/dev/block/253:50/dm/uuid
2015-12-17 11:55:17,040.040 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 2 /dev/vdd
2015-12-17 11:55:17,047.047 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:list_dev(dev = /dev/vdd2, ptype = 45b0969e-9b03-4f30-b4c6-b4b80ceff106)
2015-12-17 11:55:17,047.047 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:list_devices: [{'path': '/dev/vda', 'partitions': [{'dmcrypt': {}, 'uuid': '62088de7-53a9-4dc7-93b1-32baf094c096', 'mount': '/', 'ptype': '0fc63daf-8483-4772-8e79-3d69d8477de4', 'is_partition': True, 'fs_type': 'xfs', 'path': '/dev/vda1', 'type': 'other'}]}, {'path': '/dev/vdb', 'type': 'other', 'dmcrypt': {}, 'ptype': 'unknown', 'is_partition': False}, {'path': '/dev/vdc', 'partitions': [{'magic': 'ceph osd volume v026', 'dmcrypt': {}, 'uuid': '9ad0073e-a4b4-11e5-b02e-fa163e8241b5', 'mount': None, 'ptype': '4fbd7e29-9d25-41b8-afd0-062c0ceff05d', 'is_partition': True, 'cluster': 'ceph', 'state': 'prepared', 'fs_type': 'xfs', 'ceph_fsid': '7ca6c79d-e401-4b68-ad0f-f3b44f181b1e', 'path': '/dev/vdc1', 'type': 'data', 'whoami': '3', 'journal_dev': '/dev/vdd2', 'journal_uuid': '08d91923-0d1a-4be2-a715-cea4c9068d3f'}]}, {'path': '/dev/vdd', 'partitions': [{'dmcrypt': {}, 'uuid': '8c61499d-457f-4a29-ae1c-be361940bfcf', 'ptype': '45b0969e-9b03-4f30-b4c6-b4b80ceff106', 'is_partition': True, 'path': '/dev/vdd1', 'type': 'journal'}, {'dmcrypt': {}, 'uuid': '08d91923-0d1a-4be2-a715-cea4c9068d3f', 'ptype': '45b0969e-9b03-4f30-b4c6-b4b80ceff106', 'is_partition': True, 'journal_for': '/dev/vdc1', 'path': '/dev/vdd2', 'type': 'journal'}]}]
2015-12-17 11:55:17,129.129 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:Traceback (most recent call last):
2015-12-17 11:55:17,130.130 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:File "/usr/sbin/ceph-disk", line 4057, in <module>
2015-12-17 11:55:17,131.131 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:main(sys.argv[1:])
2015-12-17 11:55:17,131.131 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:File "/usr/sbin/ceph-disk", line 4011, in main
2015-12-17 11:55:17,132.132 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:args.func(args)
2015-12-17 11:55:17,132.132 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:File "/usr/sbin/ceph-disk", line 2841, in main_destroy
2015-12-17 11:55:17,132.132 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:base_dev = get_partition_base(dev_path)
2015-12-17 11:55:17,132.132 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:File "/usr/sbin/ceph-disk", line 611, in get_partition_base
2015-12-17 11:55:17,132.132 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:if not stat.S_ISBLK(os.lstat(dev).st_mode):
2015-12-17 11:55:17,133.133 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:OSError: [Errno 2] No such file or directory: '/dev/vdc1'
2015-12-17 11:55:17,158.158 INFO:tasks.workunit.client.0.target167114233028.stdout:../../../workunit.client.0/ceph-disk/ceph-disk-test.py::TestCephDisk::test_activate_two_separated_journal FAILED
Actions
Copy link
 #5

Updated by Loïc Dachary over 8 years ago

  • Related to Bug #14080: ceph-disk: use blkid instead of sgdisk -i added
Actions
Copy link
 #6

Updated by Loïc Dachary over 8 years ago

It turns out that sgdisk 0.8.6 -i 2 /dev/vdb removes partitions and re-adds them on CentOS 7 with a 3.10.0-229.11.1.el7 kernel, in the same way partprobe does. It is used intensively by ceph-disk and inevitably leads to races where a device temporarily disapears. The same command (sgdisk 0.8.8) on Ubuntu 14.04 with a 3.13.0-62-generic kernel only generates two udev change events and does not remove / add partitions. The source code between sgdisk 0.8.6 and sgdisk 0.8.8 did not change in a significant way and the output of strace -e ioctl sgdisk -i 2 /dev/vdb is identical in both environments.

ioctl(3, BLKGETSIZE, 20971520)          = 0
ioctl(3, BLKGETSIZE64, 10737418240)     = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, HDIO_GETGEO, {heads=16, sectors=63, cylinders=16383, start=0}) = 0
ioctl(3, HDIO_GETGEO, {heads=16, sectors=63, cylinders=16383, start=0}) = 0
ioctl(3, BLKGETSIZE, 20971520)          = 0
ioctl(3, BLKGETSIZE64, 10737418240)     = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKGETSIZE, 20971520)          = 0
ioctl(3, BLKGETSIZE64, 10737418240)     = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0

This leads me to the conclusion that the difference is in how the kernel reacts to these ioctl.

Actions
Copy link
 #7

Updated by Loïc Dachary over 8 years ago

  • Backport set to infernalis
Actions
Copy link
 #8

Updated by Loïc Dachary over 8 years ago

  • Status changed from 12 to Duplicate
  • Backport deleted (infernalis)
Actions
Copy link
 #9

Updated by Loïc Dachary over 8 years ago

  • Related to deleted (Bug #14080: ceph-disk: use blkid instead of sgdisk -i)
Actions
Copy link
 #10

Updated by Loïc Dachary over 8 years ago

  • Related to Bug #14080: ceph-disk: use blkid instead of sgdisk -i added
Actions
Copy link
 #11

Updated by Loïc Dachary over 8 years ago

The disappearance of /dev/vdb1 is a side effect of sgdisk -i indirectly triggering the equivalent of partprobe (which removes devices).

Actions
Copy link

Also available in: Atom PDF