Ceph 集群部署

环境准备

1，准备四台ubuntu1404 server 

192.168.3.106   client （部署ceph-deploy）

192.168.3.8     node1  （部署mon,mds）

192.168.3.9     node2  （部署osd)

192.168.3.10   node3   (部署osd)

2，为每台配置hosts

root@node2:~# cat /etc/hosts
127.0.0.1       localhost
192.168.3.8     node1
192.168.3.9     node2
192.168.3.10    node3  
192.168.3.106   client 

3，开启root 用户ssh

vi /etc/ssh/sshd_config

修改PermitRootLogin yes 

执行service ssh restart  重启ssh服务

4，配置client 到各个节点的无密码登录

node1,node2,node3 执行

ssh-keygen

client执行：

ssh-keygen

ssh-copy-id node1

ssh-copy-id node2

ssh-copy-id node3

root@client:~# ssh-copy-id node1
The authenticity of host 'node1 (192.168.3.8)' can't be established.
ECDSA key fingerprint is 1b:16:32:16:e8:92:4e:f2:8b:98:1f:1a:9b:e4:27:e9.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@osd's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'node1'"
and check to make sure that only the key(s) you wanted were added.

ssh mon ,登录成功！

root@client:~# ssh node1
Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.19.0-25-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Sun Jan 10 11:09:47 CST 2016

  System load:  0.01              Processes:           93
  Usage of /:   19.7% of 5.51GB   Users logged in:     1
  Memory usage: 5%                IP address for eth0: 192.168.3.8
  Swap usage:   0%

  Graph this data and manage this system at:
    https://landscape.canonical.com/

Last login: Sun Jan 10 10:34:52 2016
root@node1:~# 

安装ceph 集群

执行：apt-get install ceph-deploy 

2，配置一个mon 节点 node

执行ceph-deploy new node1 ，这里必须用主机名

root@client:~/ceph# ceph-deploy new node1
[ceph_deploy.cli][INFO  ] Invoked (1.4.0): /usr/bin/ceph-deploy new node1
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][DEBUG ] Resolving host node1
[ceph_deploy.new][DEBUG ] Monitor node1 at 192.168.3.8
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[node1][DEBUG ] connected to host: client 
[node1][INFO  ] Running command: ssh -CT -o BatchMode=yes node1
[ceph_deploy.new][DEBUG ] Monitor initial members are ['node1']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['192.168.3.8']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...

执行成功后目录下回生成 如下三个文件

ceph.mon.keyring ，ceph.conf，ceph.log

编辑ceph.conf 添加 osd_pool_default_size = 2

root@client:~/ceph# vi ceph.conf 

[global]
fsid = 3a562301-bd64-45c5-aaa0-ef57e3dfd76f
mon_initial_members = node1
mon_host = 192.168.3.8
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
osd_pool_default_size = 2

执行ceph-deploy install client node1 node2 node3

root@client:~/ceph# ceph-deploy install client node1 node2 node3
[ceph_deploy.cli][INFO  ] Invoked (1.4.0): /usr/bin/ceph-deploy install client node1 node2 node3
[ceph_deploy.install][DEBUG ] Installing stable version emperor on cluster ceph hosts client node1 node2 node3
[ceph_deploy.install][DEBUG ] Detecting platform for host client ...
[client][DEBUG ] connected to host: client 
[client][DEBUG ] detect platform information from remote host
[client][DEBUG ] detect machine type
[ceph_deploy.install][INFO  ] Distro info: Ubuntu 14.04 trusty
[client][INFO  ] installing ceph on client
[client][INFO  ] Running command: env DEBIAN_FRONTEND=noninteractive apt-get -q install --assume-yes ca-certificates
[client][DEBUG ] Reading package lists...
[client][DEBUG ] Building dependency tree...

[node3][DEBUG ] Unpacking ceph-mds (0.80.10-0ubuntu1.14.04.3) ...
[node3][DEBUG ] Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
[node3][DEBUG ] Processing triggers for ureadahead (0.100.0-16) ...
[node3][DEBUG ] ureadahead will be reprofiled on next reboot
[node3][DEBUG ] Setting up ceph-fs-common (0.80.10-0ubuntu1.14.04.3) ...
[node3][DEBUG ] Setting up ceph-mds (0.80.10-0ubuntu1.14.04.3) ...
[node3][DEBUG ] ceph-mds-all start/running
[node3][DEBUG ] Processing triggers for ureadahead (0.100.0-16) ...
[node3][INFO  ] Running command: ceph --version
[node3][DEBUG ] ceph version 0.80.10 (ea6c958c38df1216bf95c927f143d8b13c4a9e70)
Unhandled exception in thread started by 
sys.excepthook is missing
lost sys.stderr

root@node2:~# ceph --version
ceph version 0.80.10 (ea6c958c38df1216bf95c927f143d8b13c4a9e70)

激活监控节点

root@client:~/ceph# ceph-deploy mon create-initial
[ceph_deploy.cli][INFO  ] Invoked (1.4.0): /usr/bin/ceph-deploy mon create-initial
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts node1
[ceph_deploy.mon][DEBUG ] detecting platform for host node1 ...
[node1][DEBUG ] connected to host: node1 
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO  ] distro info: Ubuntu 14.04 trusty
[node1][DEBUG ] determining if provided host has same hostname in remote
[node1][DEBUG ] get remote short hostname
[node1][DEBUG ] deploying mon to node1
[node1][DEBUG ] get remote short hostname
[node1][DEBUG ] remote hostname: node1
[node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node1][DEBUG ] create the mon path if it does not exist
[node1][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-node1/done
[node1][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-node1/done
[node1][INFO  ] creating keyring file: /var/lib/ceph/tmp/ceph-node1.mon.keyring
[node1][DEBUG ] create the monitor keyring file
[node1][INFO  ] Running command: ceph-mon --cluster ceph --mkfs -i node1 --keyring /var/lib/ceph/tmp/ceph-node1.mon.keyring
[node1][DEBUG ] ceph-mon: mon.noname-a 192.168.3.8:6789/0 is local, renaming to mon.node1
[node1][DEBUG ] ceph-mon: set fsid to 3a562301-bd64-45c5-aaa0-ef57e3dfd76f
[node1][DEBUG ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-node1 for mon.node1
[node1][INFO  ] unlinking keyring file /var/lib/ceph/tmp/ceph-node1.mon.keyring
[node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[node1][DEBUG ] create the init path if it does not exist
[node1][DEBUG ] locating the `service` executable...
[node1][INFO  ] Running command: initctl emit ceph-mon cluster=ceph id=node1
[node1][INFO  ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node1.asok mon_status
[node1][DEBUG ] ********************************************************************************
[node1][DEBUG ] status for monitor: mon.node1
[node1][DEBUG ] {
[node1][DEBUG ]   "election_epoch": 2, 
[node1][DEBUG ]   "extra_probe_peers": [], 
[node1][DEBUG ]   "monmap": {
[node1][DEBUG ]     "created": "0.000000", 
[node1][DEBUG ]     "epoch": 1, 
[node1][DEBUG ]     "fsid": "3a562301-bd64-45c5-aaa0-ef57e3dfd76f", 
[node1][DEBUG ]     "modified": "0.000000", 
[node1][DEBUG ]     "mons": [
[node1][DEBUG ]       {
[node1][DEBUG ]         "addr": "192.168.3.8:6789/0", 
[node1][DEBUG ]         "name": "node1", 
[node1][DEBUG ]         "rank": 0
[node1][DEBUG ]       }
[node1][DEBUG ]     ]
[node1][DEBUG ]   }, 
[node1][DEBUG ]   "name": "node1", 
[node1][DEBUG ]   "outside_quorum": [], 
[node1][DEBUG ]   "quorum": [
[node1][DEBUG ]     0
[node1][DEBUG ]   ], 
[node1][DEBUG ]   "rank": 0, 
[node1][DEBUG ]   "state": "leader", 
[node1][DEBUG ]   "sync_provider": []
[node1][DEBUG ] }
[node1][DEBUG ] ********************************************************************************
[node1][INFO  ] monitor: mon.node1 is running
[node1][INFO  ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node1.asok mon_status
[ceph_deploy.mon][INFO  ] processing monitor mon.node1
[node1][DEBUG ] connected to host: node1 
[node1][INFO  ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node1.asok mon_status
[ceph_deploy.mon][INFO  ] mon.node1 monitor has reached quorum!
[ceph_deploy.mon][INFO  ] all initial monitors are running and have formed quorum
[ceph_deploy.mon][INFO  ] Running gatherkeys...
[ceph_deploy.gatherkeys][DEBUG ] Checking node1 for /etc/ceph/ceph.client.admin.keyring
[node1][DEBUG ] connected to host: node1 
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][DEBUG ] Got ceph.client.admin.keyring key from node1.
[ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
[ceph_deploy.gatherkeys][DEBUG ] Checking node1 for /var/lib/ceph/bootstrap-osd/ceph.keyring
[node1][DEBUG ] connected to host: node1 
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-osd.keyring key from node1.
[ceph_deploy.gatherkeys][DEBUG ] Checking node1 for /var/lib/ceph/bootstrap-mds/ceph.keyring
[node1][DEBUG ] connected to host: node1 
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-mds.keyring key from node1.

ceph.bootstrap-mds.keyring，ceph.bootstrap-osd.keyring，ceph.client.admin.keyring

root@client:~/ceph# ll
total 100
drwxr-xr-x 2 root root  4096 Jan 12 14:07 ./
drwx------ 5 root root  4096 Jan 12 13:53 ../
-rw-r--r-- 1 root root    71 Jan 12 14:07 ceph.bootstrap-mds.keyring
-rw-r--r-- 1 root root    71 Jan 12 14:07 ceph.bootstrap-osd.keyring
-rw-r--r-- 1 root root    63 Jan 12 14:07 ceph.client.admin.keyring
-rw-r--r-- 1 root root   251 Jan 12 13:53 ceph.conf
-rw-r--r-- 1 root root 66652 Jan 12 14:07 ceph.log
-rw-r--r-- 1 root root    73 Jan 12 13:52 ceph.mon.keyring

并且mon 节点 node1 上会有ceph_mon 进程

root@node1:~# ps aux|grep ceph
root      1518  0.2  1.8 154700 19284 ?        Ssl  14:48   0:00 /usr/bin/ceph-mon --cluster=ceph -i node1 -f
root      1846  0.0  0.2  11748  2208 pts/0    S+   14:49   0:00 grep --color=auto ceph

配置osd 节点

给node2,node3 分别添加一块硬盘8G,创建分区/dev/sdb1,格式化为xfs 文件系统 ，

root@node3:~# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xe5f14ca9.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): p

Disk /dev/sdb: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders, total 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xe5f14ca9

   Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1):    
Using default value 1
First sector (2048-16777215, default 2048): 2048
Last sector, +sectors or +size{K,M,G} (2048-16777215, default 16777215): 16000000

Command (m for help): p

Disk /dev/sdb: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders, total 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xe5f14ca9

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048    16000000     7998976+  83  Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
root@node3:~# mkfs.xfs /dev/sdb1
meta-data=/dev/sdb1              isize=256    agcount=4, agsize=499936 blks
         =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=1999744, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
root@node3:~# 

执行ceph-deploy osd prepare node2:/dev/sdb1 node3:/dev/sdb1

root@client:~/ceph# ceph-deploy osd prepare node2:/dev/sdb1
[ceph_deploy.cli][INFO  ] Invoked (1.4.0): /usr/bin/ceph-deploy osd prepare node2:/dev/sdb1
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks node2:/dev/sdb1:
[node2][DEBUG ] connected to host: node2 
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[ceph_deploy.osd][INFO  ] Distro info: Ubuntu 14.04 trusty
[ceph_deploy.osd][DEBUG ] Deploying osd to node2
[node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node2][INFO  ] Running command: udevadm trigger --subsystem-match=block --action=add
[ceph_deploy.osd][DEBUG ] Preparing host node2 disk /dev/sdb1 journal None activate False
[node2][INFO  ] Running command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdb1
[node2][WARNIN] Error: Partition(s) 1 on /dev/sdb1 have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use.  As a result, the old partition(s) will remain in use.  You should reboot now before making further changes.
[node2][DEBUG ] meta-data=/dev/sdb1              isize=2048   agcount=4, agsize=499936 blks
[node2][DEBUG ]          =                       sectsz=512   attr=2, projid32bit=0
[node2][DEBUG ] data     =                       bsize=4096   blocks=1999744, imaxpct=25
[node2][DEBUG ]          =                       sunit=0      swidth=0 blks
[node2][DEBUG ] naming   =version 2              bsize=4096   ascii-ci=0
[node2][DEBUG ] log      =internal log           bsize=4096   blocks=2560, version=2
[node2][DEBUG ]          =                       sectsz=512   sunit=0 blks, lazy-count=1
[node2][DEBUG ] realtime =none                   extsz=4096   blocks=0, rtextents=0
[ceph_deploy.osd][DEBUG ] Host node2 is now ready for osd use.
Unhandled exception in thread started by 
sys.excepthook is missing
lost sys.stderr

执行 ceph-deploy osd activate node2:/dev/sdb1 node3:/dev/sdb1

root@client:~/ceph# ceph-deploy osd activate node2:/dev/sdb1 node3:/dev/sdb1
[ceph_deploy.cli][INFO  ] Invoked (1.4.0): /usr/bin/ceph-deploy osd activate node2:/dev/sdb1 node3:/dev/sdb1
[ceph_deploy.osd][DEBUG ] Activating cluster ceph disks node2:/dev/sdb1: node3:/dev/sdb1:
[node2][DEBUG ] connected to host: node2 
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[ceph_deploy.osd][INFO  ] Distro info: Ubuntu 14.04 trusty
[ceph_deploy.osd][DEBUG ] activating host node2 disk /dev/sdb1
[ceph_deploy.osd][DEBUG ] will use init type: upstart
[node2][INFO  ] Running command: ceph-disk-activate --mark-init upstart --mount /dev/sdb1
[node2][WARNIN] got monmap epoch 1
[node2][WARNIN] 2016-01-12 15:26:22.781005 7f1967a62800 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
[node2][WARNIN] 2016-01-12 15:26:23.132088 7f1967a62800 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
[node2][WARNIN] 2016-01-12 15:26:23.136228 7f1967a62800 -1 filestore(/var/lib/ceph/tmp/mnt.LRhdRc) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
[node2][WARNIN] 2016-01-12 15:26:23.386713 7f1967a62800 -1 created object store /var/lib/ceph/tmp/mnt.LRhdRc journal /var/lib/ceph/tmp/mnt.LRhdRc/journal for osd.0 fsid 3a562301-bd64-45c5-aaa0-ef57e3dfd76f
[node2][WARNIN] 2016-01-12 15:26:23.386942 7f1967a62800 -1 auth: error reading file: /var/lib/ceph/tmp/mnt.LRhdRc/keyring: can't open /var/lib/ceph/tmp/mnt.LRhdRc/keyring: (2) No such file or directory
[node2][WARNIN] 2016-01-12 15:26:23.387056 7f1967a62800 -1 created new key in keyring /var/lib/ceph/tmp/mnt.LRhdRc/keyring
[node2][WARNIN] added key for osd.0
[node3][DEBUG ] connected to host: node3 
[node3][DEBUG ] detect platform information from remote host
[node3][DEBUG ] detect machine type
[ceph_deploy.osd][INFO  ] Distro info: Ubuntu 14.04 trusty
[ceph_deploy.osd][DEBUG ] activating host node3 disk /dev/sdb1
[ceph_deploy.osd][DEBUG ] will use init type: upstart
[node3][INFO  ] Running command: ceph-disk-activate --mark-init upstart --mount /dev/sdb1
[node3][WARNIN] got monmap epoch 1
[node3][WARNIN] 2016-01-12 15:26:27.482403 7f5efe4de800 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
[node3][WARNIN] 2016-01-12 15:26:28.286329 7f5efe4de800 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
[node3][WARNIN] 2016-01-12 15:26:28.416139 7f5efe4de800 -1 filestore(/var/lib/ceph/tmp/mnt.l2aD7V) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
[node3][WARNIN] 2016-01-12 15:26:29.444301 7f5efe4de800 -1 created object store /var/lib/ceph/tmp/mnt.l2aD7V journal /var/lib/ceph/tmp/mnt.l2aD7V/journal for osd.1 fsid 3a562301-bd64-45c5-aaa0-ef57e3dfd76f
[node3][WARNIN] 2016-01-12 15:26:29.444558 7f5efe4de800 -1 auth: error reading file: /var/lib/ceph/tmp/mnt.l2aD7V/keyring: can't open /var/lib/ceph/tmp/mnt.l2aD7V/keyring: (2) No such file or directory
[node3][WARNIN] 2016-01-12 15:26:29.444832 7f5efe4de800 -1 created new key in keyring /var/lib/ceph/tmp/mnt.l2aD7V/keyring
[node3][WARNIN] added key for osd.1
Unhandled exception in thread started by 
sys.excepthook is missing
lost sys.stderr

执行 ps aux | grep ceph-osd|grep -v grep

root@node2:~# ps aux | grep ceph-osd|grep -v grep
root      1752  3.9  2.9 518216 29572 ?        Ssl  15:26   0:03 /usr/bin/ceph-osd --cluster=ceph -i 0 -f
root@node2:~# 

root@node2:~# df -TH
Filesystem                      Type      Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu1404--vg-root ext4      6.0G  1.4G  4.3G  24% /
none                            tmpfs     4.1k     0  4.1k   0% /sys/fs/cgroup
udev                            devtmpfs  510M  4.1k  510M   1% /dev
tmpfs                           tmpfs     105M  476k  104M   1% /run
none                            tmpfs     5.3M     0  5.3M   0% /run/lock
none                            tmpfs     521M     0  521M   0% /run/shm
none                            tmpfs     105M     0  105M   0% /run/user
/dev/sda1                       ext2      247M   40M  195M  17% /boot
/dev/sdb1                       xfs       8.2G  5.5G  2.8G  67% /var/lib/ceph/osd/ceph-0

添加 /dev/sdb1              /var/lib/ceph/osd/ceph-0                 xfs    defaults        0 0

root@node2:~# vi /etc/fstab

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/dev/mapper/ubuntu1404--vg-root /               ext4    errors=remount-ro 0       1
# /boot was on /dev/sda1 during installation
UUID=54b04b5b-52ce-485f-95ef-70c667cfd8b3 /boot           ext2    defaults        0       2
/dev/mapper/ubuntu1404--vg-swap_1 none            swap    sw              0       0
/dev/sdb1               /var/lib/ceph/osd/ceph-0                   xfs    defaults        0 0
~

在node1 上添加 metadata server

执行ceph-deploy mds create node1

root@client:~/ceph# ceph-deploy mds create node1
[ceph_deploy.cli][INFO  ] Invoked (1.4.0): /usr/bin/ceph-deploy mds create node1
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node1:node1
[node1][DEBUG ] connected to host: node1 
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: Ubuntu 14.04 trusty
[ceph_deploy.mds][DEBUG ] remote host will use upstart
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node1
[node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node1][DEBUG ] create path if it doesn't exist
[node1][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node1 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node1/keyring
[node1][INFO  ] Running command: initctl emit ceph-mds cluster=ceph id=node1
Unhandled exception in thread started by 
sys.excepthook is missing
lost sys.stderr

root@node1:~# ps aux|grep mds
root      2025  0.1  1.6 165288 16280 ?        Ssl  15:34   0:00 /usr/bin/ceph-mds --cluster=ceph -i node1 -f
root      2046  0.0  0.2  11748  2216 pts/0    S+   15:36   0:00 grep --color=auto mds

root@node1:~# ceph osd pool create cephfs_data 10
pool 'cephfs_data' created
root@node1:~# ceph osd pool create cephfs_metadata 10
pool 'cephfs_metadata' created
root@node1:~# ceph fs new cephfs cephfs_metadata cephfs_data
no valid command found; 10 closest matches:
fsid
Error EINVAL: invalid command
root@node1:~# ceph fs ls
no valid command found; 10 closest matches:
fsid
Error EINVAL: invalid command
root@node1:~# ceph mds stat
e4: 1/1/1 up {0=node1=up:active}
root@node1:~# 

执行 ceph-deploy admin node1 node2 node3

root@client:~/ceph# ceph-deploy admin node1 node2 node3
[ceph_deploy.cli][INFO  ] Invoked (1.4.0): /usr/bin/ceph-deploy admin node1 node2 node3
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node1
[node1][DEBUG ] connected to host: node1 
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] get remote short hostname
[node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node2
[node2][DEBUG ] connected to host: node2 
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[node2][DEBUG ] get remote short hostname
[node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node3
[node3][DEBUG ] connected to host: node3 
[node3][DEBUG ] detect platform information from remote host
[node3][DEBUG ] detect machine type
[node3][DEBUG ] get remote short hostname
[node3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

root@node1:~# ceph health
HEALTH_OK

使用rbd

新建一个ceph pool  
rados mkpool test
在pool中新建一个镜像（ip 为mon节点）
rbd create test-1 --size 4096 -p test -m 172.17.106.122
把镜像映射到pool块设备中
rbd map test-1 -p test --name client.admin
查看rbd的映射关系
rbd showmapped

root@node1:~# rados mkpool test
successfully created pool test
root@node1:~# lsmod |grep rbd
root@node1:~# rbd create test-1 --size 4096 -p test -m 192.168.3.8
root@node1:~# rbd map test-1 -p test --name client.admin
root@node1:~# rbd showmapped
id pool image  snap device    
0  test test-1 -    /dev/rbd0

root@node1:~# mkfs.ext4 -m0 /dev/rbd0
mke2fs 1.42.9 (4-Feb-2014)
Discarding device blocks: done                            
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=1024 blocks, Stripe width=1024 blocks
262144 inodes, 1048576 blocks
0 blocks (0.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1073741824
32 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

root@node1:~# mount /dev/rbd0 /mnt

root@node1:~# df -TH
Filesystem                      Type      Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu1404--vg-root ext4      6.0G  1.4G  4.3G  24% /
none                            tmpfs     4.1k     0  4.1k   0% /sys/fs/cgroup
udev                            devtmpfs  510M  4.1k  510M   1% /dev
tmpfs                           tmpfs     105M  472k  104M   1% /run
none                            tmpfs     5.3M     0  5.3M   0% /run/lock
none                            tmpfs     521M     0  521M   0% /run/shm
none                            tmpfs     105M     0  105M   0% /run/user
/dev/sda1                       ext2      247M   40M  195M  17% /boot
/dev/rbd0                       ext4      4.1G  8.4M  4.1G   1% /mnt