前言

已经实施过2次Hadoop大数据平台的基础环境搭建,对于硬盘的RAID模式都不约而同选择了JBOD,毕竟大数据对于文件系统的管理本身就是基于分布式,由至少3份副本来保障数据可靠性,只是不做RAID后的SAS硬盘读写性能就有点惨不忍睹。这次Hadoop大数据平台工具选择的是Cloudera,大家感兴趣的化也可以浏览下它的官网。本篇文章主要介绍如何实现在不停机状态下在线更换JBOD硬盘。

利用StorCLI命令不停机在线更换JBOD硬盘

更新历史

2016年12月31日 - 初稿

阅读原文 - https://wsgzao.github.io/post/jbod/

扩展阅读

Cloudera - http://www.cloudera.com/


JBOD原理

JBOD ( Just a Bunch Of Disks )不是标准的 RAID 等级,它通常用来表示一个没有控制软件提供协调控制的磁盘集合。 JBOD 将多个物理磁盘串联起来,提供一个巨大的逻辑磁盘。 JBOD 的数据存放机制是由第一块磁盘开始按顺序往后存储,当前磁盘存储空间用完后,再依次往后面的磁盘存储数据。 JBOD 存储性能完全等同于单块磁盘,而且也不提供数据安全保护。它只是简单提供一种扩展存储空间的机制, JBOD 可用存储容量等于所有成员磁盘的存储空间之和。目前 JBOD 常指磁盘柜,而不论其是否提供 RAID 功能。

RAID磁盘阵列配置和调优小结 - https://wsgzao.github.io/post/raid/

参考文档和工具

StorCLI_RefMan_revf
https://www.broadcom.com/support/download-search/?dk=StorCLI

StorCLI (Command Line) Utility for Storage Management v1.07.07 on Linux
https://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5092950

使用方式

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
#查询硬件RAID卡信息
[root@localhost Linux]# cat /proc/scsi/scsi |more

Attached devices:
Host: scsi0 Channel: 02 Id: 00 Lun: 00
Vendor: IBM Model: ServeRAID M5210 Rev: 4.62
Type: Direct-Access ANSI SCSI revision: 05

#解压安装storcli
unzip ibm_utl_sraidmr_storcli-1.07.07_linux_32-64.zip
rpm -Uvh storcli-1.07.07-1.noarch.rpm

#设置软连接
ln -s /opt/MegaRAID/storcli/storcli64 /bin/storcli
ln -s /opt/MegaRAID/storcli/storcli64 /sbin/storcli

#Raid数量及当前状态
[root@localhost Linux]# storcli show ctrlcount
Status Code = 0
Status = Success
Description = None

Controller Count = 2

#查询虚拟磁盘大小,raid结构等
[root@localhost Linux]# storcli /c0 /v0 show
Controller = 0
Status = Success
Description = None


Virtual Drives :
==============

-----------------------------------------------------------
DG/VD TYPE State Access Consist Cache sCC Size Name
-----------------------------------------------------------
0/0 RAID1 Optl RW Yes RWTD - 278.465 GB
-----------------------------------------------------------

Cac=CacheCade|Rec=Recovery|OfLn=OffLine|Pdgd=Partially Degraded|dgrd=Degraded
Optl=Optimal|RO=Read Only|RW=Read Write|B=Blocked|Consist=Consistent|
R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack|
AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled
Check Consistency
#可以看出此为RAID1,一个虚拟磁盘,大小为278.465 GB

#查看control0的物理硬盘
[root@localhost Linux]# storcli /c0 /eall /sall show
Controller = 0
Status = Success
Description = Show Drive Information Succeeded.


Drive Information :
=================

-------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp
-------------------------------------------------------------------------
252:16 8 Onln 0 278.465 GB SAS HDD N Y 512B ST300MP0035 X U
252:17 9 Onln 0 278.465 GB SAS HDD N Y 512B ST300MP0035 X U
-------------------------------------------------------------------------

EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare
UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
SeSz-Sector Size|Sp-Spun|U-Up|D-Down|T-Transition|F-Foreign
UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded
CFShld-Configured shielded

#可以看到总计有两颗硬盘,每一颗大小278.465GB,是否在线等

#查看control1的物理硬盘
[root@localhost Linux]# storcli /c1 /eall /sall show
Controller = 1
Status = Success
Description = Show Drive Information Succeeded.


Drive Information :
=================

------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp
------------------------------------------------------------------------
8:0 23 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:1 24 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:2 22 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:3 19 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:4 13 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:5 14 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:6 18 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:7 12 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:8 10 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:9 16 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:10 11 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:11 21 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:12 15 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:13 9 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
------------------------------------------------------------------------

EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare
UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
SeSz-Sector Size|Sp-Spun|U-Up|D-Down|T-Transition|F-Foreign
UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded
CFShld-Configured shielded

#可以看到总计有两颗硬盘,每一颗大小1.0810 TB,是否在线等


#设置硬盘状态
storcli /c1/e8/s0 show all

storcli /c1/e8/s0 start locate
storcli /c1/e8/s0 set offline
storcli /c1/e8/s0 set online
storcli /c1/e8/s0 set jbod


StorCLI Help

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
#查看StorCLI帮助信息
[root@localhost Linux]# storcli -h
Storage Command Line Tool Ver 1.07.07 Nov 15, 2013

(c)Copyright 2013, LSI Corporation, All Rights Reserved.


storcli -v
storcli -h| -help| ?
storcli -h| -help| ? legacy
storcli show
storcli show all
storcli show ctrlcount
storcli /cx add vd r[0|1|5|6|00|10|50|60]
[Size=<VD1_Sz>,<VD2_Sz>,..|all] [name=<VDNAME1>,..]
drives=e:s|e:s-x|e:s-x,y,e:s-x,y,z [PDperArray=x][SED]
[pdcache=on|off|default][pi][DimmerSwitch(ds)=default|automatic(auto)|
none|maximum(max)|MaximumWithoutCaching(maxnocache)]
[wt|wb|awb][nora|ra] [direct|cached] [cachevd]
[Strip=<8|16|32|64|128|256|512|1024>] [AfterVd=X]
[Spares = [e:]s|[e:]s-x|[e:]s-x,y] [force][ExclusiveAccess]
storcli /cx add vd each r0 [name=<VDNAME1>,..] [drives=e:s|e:s-x|e:s-x,y]
[SED] [pdcache=on|off|default][pi] [DimmerSwitch(ds)=default|automatic(auto)|
none|maximum(max)|MaximumWithoutCaching(maxnocache)]
[wt|wb|awb] [nora|ra] [direct|cached]
[Strip=<8|16|32|64|128|256|512|1024>] [ExclusiveAccess]
storcli /cx add VD cachecade|nytrocache r[aid][0,1,10, 1EC]
drives = [e:]s|[e:]s-x|[e:]s-x,y [WT| WB] [assignvds = 0,1,2] [BOOTVOLSIZE=x]
storcli /cx/ex show all
storcli /cx/ex show status
storcli /cx/vx del [cachecade] [force]
storcli /cx delete events
storcli /cx show events [[type= <sincereboot| sinceshutdown| includedeleted|
latest=x| ccincon vd=<0,1,...>] filter=<info|warning|critical|fatal>]
file=<filepath>
storcli /cx show eventloginfo
storcli /cx show health [all]
storcli /cx delete securitykey
storcli /cx set securitykey=xxxxxxxx {passphrase=xxxx} {keyid=xxx}
storcli /cx start Diag Duration=<Val>
storcli /cx set securitykey keyid=xxx
storcli /cx compare securitykey=xxxxxxxxxx
storcli /cx set termlog[=on|off|offthisboot]
storcli /cx show termlog [type=config|contents]
storcli /cx delete termlog
storcli /cx Compare factorysettings file=<filepath>
storcli /cx Compare factorysettings raw file=<filepath>
storcli /cx set securitykey=xxxxxxxx oldsecuritykey=xxxxxxxx
{passphrase=xxxx} {keyid=xxx}
storcli /cx/dx show
storcli /cx/dall show cachecade
storcli /cx/dx show all
storcli /cx show freespace
storcli /cx/fall show [all] [securityKey = xxx]
storcli /cx/fall del|delete [securityKey = xxx]
storcli /cx/fall import [preview] [securityKey = xxx]
storcli /cx/vx set ssdcaching=on|off
storcli /cx/dx set security=on
storcli /cx/vx show expansion
storcli /cx show fshinting
storcli /cx/vx set fshinting=<value>
storcli /cx/vx expand Size=<xx> [expandarray]
storcli /cx[/ex]/sx show
storcli /cx[/ex]/sx show all
storcli /cx[/ex]/sx start rebuild
storcli /cx[/ex]/sx stop rebuild
storcli /cx[/ex]/sx pause rebuild
storcli /cx[/ex]/sx resume rebuild
storcli /cx[/ex]/sx show rebuild
storcli /cx[/ex]/sx show health
storcli /cx[/ex]/sx start copyback target=e:s
storcli /cx[/ex]/sx stop copyback
storcli /cx[/ex]/sx pause copyback
storcli /cx[/ex]/sx resume copyback
storcli /cx[/ex]/sx show copyback
storcli /cx[/ex]/sx start initialization
storcli /cx[/ex]/sx stop initialization
storcli /cx[/ex]/sx show initialization
storcli /cx[/ex]/sx start locate
storcli /cx[/ex]/sx stop locate
storcli /cx[/ex]/sx show securitykey keyid
storcli /cx[/ex]/sx add hotsparedrive [DGs=<N|0,1,2...>] [enclaffinity]
[nonrevertible]
storcli /cx[/ex]/sx delete hotsparedrive
storcli /cx[/ex]/sx spinup
storcli /cx[/ex]/sx spindown
storcli /cx[/ex]/sx set online
storcli /cx[/ex]/sx set offline
storcli /cx[/ex]/sx set missing
storcli /cx[/ex]/sx set jbod
storcli /cx[/ex]/sx set good [force]
storcli /cx[/ex]/sx insert dg=A array=B row=C
storcli /cx/vx set emulationType=0|1
storcli /cx/vx set wrcache=WT|WB|AWB
storcli /cx/vx set rdcache=RA|NoRA
storcli /cx/vx set iopolicy=Cached|Direct
storcli /cx/vx set accesspolicy=RW|RO|Blocked|RmvBlkd
storcli /cx/vx set pdcache=On|Off|Default
storcli /cx/vx set name=<NameString>
storcli /cx/vx set HostAccess=ExclusiveAccess|SharedAccess
storcli /cx/vx set ds=Default|Auto|None|Max|MaxNoCache
storcli /cx/vx set autobgi=On|Off
storcli /cx/vx set pi=Off
storcli /cx/vx show
storcli /cx/vx show all
storcli /cx/vx show init
storcli /cx/vx show cc
storcli /cx/vx show erase
storcli /cx/vx show migrate
storcli /cx/vx show bgi
storcli /cx/vx show autobgi
storcli /cx/vx show trim
storcli /cx set consistencycheck|cc=[off|seq|conc] [delay=value]
starttime=yyyy/mm/dd hh] [excludevd=x-y,z]
storcli /cx show cc|consistencycheck
storcli /cx show ocr
storcli /cx set ocr=<on|off>
storcli /cx/vx start init[Full][Force]
storcli /cx/vx start erase [simple|normal|thorough|standard][patternA=<val>]
[patternB=<val>]
storcli /cx/vx start cc[Force]
storcli /cx/vx start migrate type=raidx [option=add|remove
drives=[e:]s|[e:]s-x|[e:]s-x,y] [Force]
storcli /cx/vx stop init
storcli /cx/vx stop erase
storcli /cx/vx stop cc
storcli /cx/vx stop bgi
storcli /cx/vx pause cc
storcli /cx/vx pause bgi
storcli /cx/vx resume cc
storcli /cx/vx resume bgi
storcli /cx show
storcli /cx show all
storcli /cx show preservedcache
storcli /cx/vx delete preservedcache[force]
storcli /cx[/ex]/sx download src=<filepath> [satabridge] [mode= 5|7]
storcli /cx/ex download src=<filepath> [forceActivate]
storcli /cx[/ex]/sx secureerase [force]
storcli /cx[/ex]/sx start erase [simple| normal| thorough | standard| threepass]
[patternA=<val>] [patternB=<val>]
storcli /cx[/ex]/sx stop erase
storcli /cx[/ex]/sx show erase
storcli /cx[/ex]/sx show rawdata pageaddr=<pageaddress in hex> file=<filename>
storcli /cx[/ex]/sx set bootdrive=<on|off>
storcli /cx/vx set bootdrive=<on|off>
storcli /cx show bootdrive
storcli /cx show bootwithpinnedcache
storcli /cx set bootwithpinnedcache=<on|off>
storcli /cx show activityforlocate
storcli /cx set activityforlocate=<on|off>
storcli /cx show copyback
storcli /cx set copyback=<on|off> type=smartssd|smarthdd|all
storcli /cx show jbod
storcli /cx set jbod=<on|off>
storcli /cx set autorebuild=<on|off>
storcli /cx show autorebuild
storcli /cx show cachebypass
storcli /cx set cachebypass=<on|off>
storcli /cx show usefdeonlyencrypt
storcli /cx set usefdeonlyencrypt=<on|off>
storcli /cx show prcorrectunconfiguredareas
storcli /cx set prcorrectunconfiguredareas=<on|off>
storcli /cx show batterywarning
storcli /cx set batterywarning=<on|off>
storcli /cx show abortcconerror
storcli /cx set abortcconerror=<on|off>
storcli /cx show ncq
storcli /cx set ncq=<on|off>
storcli /cx show maintainpdfailhistory
storcli /cx set maintainpdfailhistory=<on|off>
storcli /cx show restorehotspare
storcli /cx set restorehotspare=<on|off>
storcli /cx set bios [state=<on|off>] [Mode=<SOE|BE|HCOE|HSM>] [abs=<on|off>]
storcli /cx show bios
storcli /cx show alarm
storcli /cx set alarm=<on|off|silence>
storcli /cx show foreignautoimport
storcli /cx set foreignautoimport=<on|off>
storcli /cx show directpdmapping
storcli /cx set directpdmapping=<on|off>
storcli /cx show rebuildrate
storcli /cx set rebuildrate=<value>
storcli /cx show loadbalancemode
storcli /cx set loadbalancemode=<on|off>
storcli /cx show eghs
storcli /cx set eghs [state=<on|off>] [eug=<on|off>] [smarter=<on|off>]
storcli /cx show cacheflushint
storcli /cx set cacheflushint=<value>
storcli /cx show prrate
storcli /cx set prrate=<value>
storcli /cx show ccrate
storcli /cx set ccrate=<value>
storcli /cx show bgirate
storcli /cx set bgirate =<value>
storcli /cx show dpm
storcli /cx set dpm =<on|off>
storcli /cx show sgpioforce
storcli /cx set sgpioforce =<on|off>
storcli /cx set supportssdpatrolread =<on|off>
storcli /cx show migraterate
storcli /cx set migraterate=<value>
storcli /cx show spinupdrivecount
storcli /cx show wbsupport
storcli /cx set spinupdrivecount=<value>
storcli /cx show spinupdelay
storcli /cx set spinupdelay=<value>
storcli /cx show coercion
storcli /cx set coercion=<value>
storcli /cx show smartpollinterval
storcli /cx set smartpollinterval=<value>
storcli /cx show eccbucketsize
storcli /cx set eccbucketsize=<value>
storcli /cx show eccbucketleakrate
storcli /cx set eccbucketleakrate=<value>
storcli /cx show backplane
storcli /cx set backplane mode=<value> expose=<on/off>
storcli /cx show perfmode
storcli /cx set perfmode=<value> [maxflushlines=<value> numiostoorder=<value>]
storcli /cx show perfmodevalues
storcli /cx show pi
storcli /cx set pi [state=<on|off>] [import=<on|off>]
storcli /cx show time
storcli /cx set time=<yyyymmdd hh:mm:ss | systemtime>
storcli /cx show ds
storcli /cx set ds=OFF type=1|2|3|4
storcli /cx set ds=ON type=1|2 [properties]
storcli /cx set ds=ON type=3|4 DefaultLdType=<val> [properties]
storcli /cx set ds [properties]
storcli /cx show safeid
storcli /cx show rehostinfo
storcli /cx show pci
storcli /cx show ASO
storcli /cx set aso key=<key value> preview
storcli /cx set aso key=<key value>
storcli /cx set aso transfertovault
storcli /cx set aso rehostcomplete
storcli /cx set aso deactivatetrialkey
storcli /cx set factory defaults
storcli /cx download file=<filepath> [fwtype=<val>] [ResetNow] [nosigchk]
[noverchk]
storcli /cx flush|flushcache
storcli /cx/px show
storcli /cx/px show all
storcli /cx/px set linkspeed=0|1.5|3|6|12
storcli /cx/bbu show
storcli /cx/bbu set SCAPVPD file=<input file path> VPDPage=<SCapVPDFixed>
storcli /cx/bbu show all
storcli /cx/bbu show status
storcli /cx/bbu show properties
storcli /cx/bbu show learn
storcli /cx/bbu show gasgauge Offset=xxxx Numbytes=n
storcli /cx/bbu start learn
storcli /cx/bbu show modes
storcli /cx/bbu set [ learnDelayInterval=<val> | bbuMode=<val>
|learnStartTime=[DDD HH | off] | autolearnmode=<val> | powermode=sleep | writeaccess=sealed ]
storcli /cx/cv show
storcli /cx/cv show all
storcli /cx/cv show status
storcli /cx/cv show learn
storcli /cx/cv start learn
storcli /cx show securitykey keyid
storcli /cx start patrolread
storcli /cx stop patrolread
storcli /cx pause patrolread
storcli /cx resume patrolread
storcli /cx show patrolRead
storcli /cx show powermonitoringinfo
storcli /cx set patrolread = {{on mode=<auto|manual> }|{off}}
storcli /cx set patrolread [starttime=< yyyy/mm/dd hh>]
[maxconcurrentpd =<value>] [includessds=<on|onlymixed|off>]
[uncfgareas=on|off]
storcli /cx set patrolread delay = <value>
storcli /cx[/ex]/sx show diag paniclog [Query] | [ExtractSlot=x]
| [EraseSlot=x] [file=filepath]
storcli /cx[/ex]/sx show diag smartlog [file=filepath]
storcli /cx[/ex]/sx show diag errorlog [file=filepath]
storcli /cx del Nytrocache [force]
storcli /cx show badblocks
storcli /cx flasherase
storcli /cx shutdown
storcli /cx/mx set mode=<Internal | External | Auto>
storcli /cx/mx show
storcli /cx transform iMR
storcli /cx restart
storcli /cx/vx show BBMT
storcli /cx/vx delete BBMT
storcli /xd add type=CacheDev
<devices=comma separated cache devices | ALL> [force]
storcli /xd add type=VirtualDrive
<devices=comma separated Virtual drives | ALL>
storcli /xd delete type=<CacheDev | VirtualDrive>
<devices=comma separated cache devices or
Virtual drives as applicable | ALL>
storcli /xd show type=<CacheDev | VirtualDrive> [state=<assigned|unassigned>]
storcli /xd show
storcli /xd show perfmon
storcli /xd/wd<X|ALL> show
storcli /xd/wd<X|ALL> show safeid
storcli /xd set online [Vds=comma separated VD ids]
storcli /xd/wdX apply key=value
storcli /cx[/ex]/sx start format [thorough]
storcli /cx show dequeuelog file=<filepath>


Note:Use 'page=[x]'as the last option in all the commands to set the page break.
X=lines per page. E.g. 'storcli help page=10'

文章目录
  1. 1. 前言
  2. 2. 更新历史
  3. 3. JBOD原理
  4. 4. 参考文档和工具
  5. 5. 使用方式
  6. 6. StorCLI Help