HP Smart Storage Admin (ssacli)

Guestbook

Check RAID Status

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[root@vmesxi04:~]  /opt/smartstorageadmin/ssacli/bin/ssacli ctrl all show config
HPE Smart Array P408i-a SR Gen10 in Slot 0 (Embedded)  (sn: PWXKV0BRHDS7ID)

   Internal Drive Cage at Port 1I, Box 3, OK
   Internal Drive Cage at Port 2I, Box 0, OK

   Port Name: 1I (Mixed)
   Port Name: 2I (Mixed)

   Array A (SAS, Unused Space: 748  MB)

      logicaldrive 1 (279.00 GB, RAID 1, OK)

      physicaldrive 1I:3:1 (port 1I:box 3:bay 1, SAS HDD, 300 GB, Predictive Failure)
      physicaldrive 1I:3:2 (port 1I:box 3:bay 2, SAS HDD, 300 GB, OK)

   SEP (Vendor ID HPE, Model Smart Adapter) 379  (WWID: 51402EC01529DCA8)

Check Disk Status

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
[root@vmesxi04:~]  /opt/smartstorageadmin/ssacli/bin/ssacli ctrl slot=0 pd all show detail

HPE Smart Array P408i-a SR Gen10 in Slot 0 (Embedded)

   Array A

      physicaldrive 1I:3:1
         Port: 1I
         Box: 3
         Bay: 1
         Status: Predictive Failure
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 300 GB
         Drive exposed to OS: False
         Logical/Physical Block Size: 512/512
         Rotational Speed: 15000
         Firmware Revision: HPD4
         Serial Number: 2050A0LDFA3F
         WWID: 50000399F819294E
         Model: HP      EH000300JWHPL
         Current Temperature (C): 38
         Maximum Temperature (C): 45
         PHY Count: 2
         PHY Transfer Rate: 12.0Gbps, Unknown
         PHY Physical Link Rate: 12.0Gbps, Unknown
         PHY Maximum Link Rate: 12.0Gbps, 12.0Gbps
         Drive Authentication Status: OK
         Carrier Application Version: 11
         Carrier Bootloader Version: 6
         Sanitize Erase Supported: True
         Sanitize Estimated Max Erase Time: 2 hour(s), 43 minute(s)
         Unrestricted Sanitize Supported: True
         Shingled Magnetic Recording Support: None
         Drive Unique ID: 50000399F819294D

      physicaldrive 1I:3:2
         Port: 1I
         Box: 3
         Bay: 2
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 300 GB
         Drive exposed to OS: False
         Logical/Physical Block Size: 512/512
         Rotational Speed: 15000
         Firmware Revision: HPD4
         Serial Number: 2090A16HFA3F
         WWID: 50000399F8306D42
         Model: HP      EH000300JWHPL
         Current Temperature (C): 34
         Maximum Temperature (C): 44
         PHY Count: 2
         PHY Transfer Rate: 12.0Gbps, Unknown
         PHY Physical Link Rate: 12.0Gbps, Unknown
         PHY Maximum Link Rate: 12.0Gbps, 12.0Gbps
         Drive Authentication Status: OK
         Carrier Application Version: 11
         Carrier Bootloader Version: 6
         Sanitize Erase Supported: True
         Sanitize Estimated Max Erase Time: 2 hour(s), 43 minute(s)
         Unrestricted Sanitize Supported: True
         Shingled Magnetic Recording Support: None
         Drive Unique ID: 50000399F8306D41

RAID rebuilding status

1
2
3
[root@suvmesxi15:~] /opt/smartstorageadmin/ssacli/bin/ssacli ctrl slot=0 ld all show status

   logicaldrive 1 (558.88 GB, RAID 1): Recovering, 8.30% complete

Task Scheduler

Guestbook

Create schedule task

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# Define task name
$TaskName = "ShowTimeTask"

# Define script path and name
$ScriptPath = "C:\temp\show_time.bat"

# Define taks path in Task Scheduler
$TaskPath = "\MyLab2"

# Define task start at 08:00 and 13:00 everyday
$AdditionalTriggerTime1 = "08:00"
$AdditionalTriggerTime2 = "13:00"

# Create a schedule at 08:00
$Action = New-ScheduledTaskAction -Execute $ScriptPath
$AdditionalTrigger1 = New-ScheduledTaskTrigger -Daily -At $AdditionalTriggerTime1

Register-ScheduledTask -Action $Action -Trigger $AdditionalTrigger1 -TaskName $TaskName -TaskPath $TaskPath

# Add another schedule at 13:00
$AdditionalTrigger2 = New-ScheduledTaskTrigger -Daily -At $AdditionalTriggerTime2
$Task = Get-ScheduledTask -TaskName $TaskName 
$Task.Triggers += $AdditionalTrigger2
Set-ScheduledTask -TaskName $TaskName -Trigger $Task.Triggers -TaskPath $TaskPath

# Add the task will be enabled at system startup
$AdditionalTrigger3 = New-ScheduledTaskTrigger -AtStartup
$Task = Get-ScheduledTask -TaskName $TaskName 
$Task.Triggers += $AdditionalTrigger3
Set-ScheduledTask -TaskName $TaskName -Trigger $Task.Triggers -TaskPath $TaskPath

# Run whether user is logged on or not
$principal = New-ScheduledTaskPrincipal -UserID "NT AUTHORITY\SYSTEM" -LogonType ServiceAccount -RunLevel Highest
Set-ScheduledTask -TaskName $TaskName -Trigger $Task.Triggers -TaskPath $TaskPath -Principal $principal

Delete a schedule task

1
2
3
4
5
# Remove a schedule task
$TaskName = "ShowTimeTask"
$Task = Get-ScheduledTask -TaskName "ShowTimeTask"

Unregister-ScheduledTask -TaskName  $TaskName -Confirm:$false

QuadStor VTL Installation and Configuration

Guestbook

Setting Linux Environment (CentOS 7.9)

1
2
3
4
5
6
7
[root@localhost ~]# hostnamectl set-hostname vtl01

[root@localhost ~]# systemctl disable firewalld --now

[root@localhost ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

[root@vtl01 ~]# timedatectl set-timezone Asia/Hong_Kong

Install Pre-Requisite of QuadStor VTL

1
2
3
4
5
6
7
8
9
10
[root@vtl01 ~]# yum install httpd make gcc perl kernel-devel e2fsprogs-libs sg3_utils iotop sysstat tar xz postgresql-libs postgresql-server chkconfig mtx

[root@vtl01 ~]# rpm -qa | grep kernel-devel
kernel-devel-3.10.0-1160.119.1.el7.x86_64

[root@vtl01 ~]# uname -r
3.10.0-1160.el7.x86_64

[root@vtl01 ~]# yum upgrade kernel
[root@vtl01 ~]# reboot

Secure Boot is not supported

1
2
[root@vtl01 ~]# grep "Secure boot enabled" /var/log/messages
<<empty>>

Enable httpd startup

1
[root@vtl01 ~]# systemctl enable httpd --now

Install the QuadStore VTL software

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@vtl01 ~]# rpm -Uivh https://www.quadstor.com/vtldownloads/quadstor-vtl-ext-3.0.79.14-rhel.x86_64.rpm
Retrieving https://www.quadstor.com/vtldownloads/quadstor-vtl-ext-3.0.79.14-rhel.x86_64.rpm
Preparing...                          ################################# [100%]
Adding group vtprocgrp
usermod -G vtprocgrp apache > /dev/null 2>&1
Updating / installing...
   1:quadstor-vtl-ext-3.0.79.14-rhel  ################################# [100%]
Performing post install. Please wait...
Created symlink from /etc/systemd/system/multi-user.target.wants/quadstorvtl.service to /usr/lib/systemd/system/quadstorvtl.service.
Building required kernel modules
Running /quadstorvtl/bin/builditf. This may take a few minutes.
Installation complete. A system reboot is required to start the VTL service

[root@vtl01 ~]# reboot

Verify QuadStor VTL status

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[root@vtl01 ~]# systemctl status quadstorvtl
● quadstorvtl.service - QUADStor Virtual Tape Library
   Loaded: loaded (/usr/lib/systemd/system/quadstorvtl.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2024-06-29 12:30:59 HKT; 1min 10s ago
  Process: 1072 ExecStart=/quadstorvtl/etc/quadstorvtl.init start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/quadstorvtl.service
           ├─2242 /quadstorvtl/sbin/coredev
           ├─6044 /quadstorvtl/sbin/ietd
           └─6055 /quadstorvtl/sbin/vtmdaemon

Jun 29 12:30:59 vtl01 vtmdaemon[6055]: tl_server_load_conf:3895 query disk check
Jun 29 12:30:59 vtl01 vtmdaemon[6055]: tl_server_load_conf:3909 trigger qload
Jun 29 12:30:59 vtl01 vtmdaemon[6055]: tl_server_load_conf:3911 ioctl qload
Jun 29 12:30:59 vtl01 vtmdaemon[6055]: configure_fc_ports_mode:3071 target 1
Jun 29 12:30:59 vtl01 vtmdaemon[6055]: tl_server_load_conf:3917 restart export jobs
Jun 29 12:30:59 vtl01 vtmdaemon[6055]: tl_server_load_conf:3920 reply to client
Jun 29 12:30:59 vtl01 vtmdaemon[6055]: tl_server_restart_export_jobs:6619 start
Jun 29 12:30:59 vtl01 vtmdaemon[6055]: tl_server_load_conf:3922 end
Jun 29 12:30:59 vtl01 vtmdaemon[6055]: tl_server_restart_export_jobs:6682 end
Jun 29 12:30:59 vtl01 systemd[1]: Started QUADStor Virtual Tape Library.

Add physical storage to QuadStor

1
2
3
4
5
6
7
8
9
10
11
12
13
14
List all disks
[root@vtl01 bin]# /quadstorvtl/bin/bdconfig -l
ID  Vendor Model   SerialNumber          Name     Pool Size     Used     Status
0   ATA    VMware  00000000000000000001  /dev/sdb N/A  50.00    0.00     N/A

[root@vtl01 bin]# dd if=/dev/zero of=/dev/sdb count=64 bs=1M

Add a disk
[root@vtl01 bin]# /quadstorvtl/bin/bdconfig -a -d /dev/sdb  -g Default
Adding disk /dev/sdb successful

[root@vtl01 bin]# /quadstorvtl/bin/bdconfig -l
ID  Vendor Model   SerialNumber          Name     Pool    Size     Used     Status
1   ATA    VMware  00000000000000000001  /dev/sdb Default 50.00    0.93     Active D

HMC useful commands

Guestbook

List all management systems and its state

1
2
3
4
5
6
7
8
9
10
hscroot@suapvhmc01:~> lssyscfg -r sys -Fname,type_model,serial_num,ipaddr,state
ssuxspr01,9009-22A,7868070,192.168.229.69,Initializing
sduxspr01,9009-22A,7868080,192.168.231.74,Operating
suuxspr06,9009-22G,7821251,192.168.233.213,Operating
suuxspr01,9009-42A,7886280,192.168.233.178,Operating
suuxspr02,9009-42A,7886210,192.168.233.180,Operating
suuxspr03,9009-42A,7886220,192.168.233.182,Operating
suuxspr04,9009-42A,7886270,192.168.233.184,Operating
stuxspr01,9009-22A,7838630,192.168.237.140,Operating
suuxspr05,9009-22G,7821261,192.168.233.226,Operating

Remove a management system from HMC

1
hscroot@suapvhmc01:~> rmsysconn -o remove --ip 192.168.229.69

Add a management system to HMC

1
hscroot@suapvhmc01:~> mksysconn --ip 192.168.229.69 -r sys -u admin --passwd admin

Power of management system

1
hscroot@suapvhmc01:~> chsysstate -m ssuxspr01  -o off -r sys --immed

List all LPARs in management system

1
2
3
4
hscroot@suapvhmc01:~> ssuxvios1a,1,Running,default_profile
ssuxvios1b,2,Running,default_profile
ssuxapps01,3,Not Activated,default_profile
ssuxiedb01,4,Not Activated,default_profile

List all LPARs state

1
2
3
4
5
6
7
8
9
hscroot@suapvhmc01:~> lssyscfg -m sduxspr01 -r lpar -F name,lpar_id,state,default_profile | sort -k 2
sduxapps01,3,Running,default_profile
sduxiedb01,7,Running,default_profile
sduxviops01a,1,Running,default_profile
sduxviops01b,2,Running,default_profile
reserve01,5,Not Activated,default_profile
reserve02,6,Not Activated,default_profile
reserve03,8,Not Activated,default_profile
reserve04,4,Not Activated,default_profile

System LED

1
2
3
4
5
6
7
hscroot@suapvhmc01:~> lsled -m suuxspr01 -r sa -t phys
state=on

hscroot@suapvhmc01:~> chled -m suuxspr01 -o off -r sa -t phys

hscroot@suapvhmc01:~> lsled -m suuxspr01 -r sa -t phys
state=off

APC Command Linne Interface (CLI

Guestbook

Access APC Smart-UPS vis ssh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
$ ssh -l apc 192.168.152.22

Schneider Electric                      Network Management Card AOS    v2.5.0.8
(c) Copyright 2023 All Rights Reserved  Smart-UPS APP                  v2.5.0.6
-------------------------------------------------------------------------------
Name      : apc6324C8                                 Date : 06/24/2024
Contact   : Unknown                                   Time : 11:21:42
Location  : Unknown                                   User : Super User
Up Time   : 0 Days 0 Hours 28 Minutes                 Stat : P+ N4+ N6+ A+
-------------------------------------------------------------------------------
IPv4               : Enabled            IPv6               : Enabled
Ping Response      : Enabled
-------------------------------------------------------------------------------
HTTP               : Enabled            HTTPS              : Enabled
FTP                : Enabled            Telnet             : Disabled
SSH/SCP            : Enabled            SNMPv1             : Disabled
SNMPv3             : Disabled           Modbus TCP         : Disabled
BACnet/IP          : Disabled
-------------------------------------------------------------------------------
Super User         : Enabled            RADIUS             : Disabled
Administrator      : 1 Enabled          Device User        : Disabled
Read-Only User     : Disabled           Network-Only User  : Disabled


Type ? for command listing
Use tcpip command for IP address(-i), subnet(-s), and gateway(-g)

* Reboot required for change to take effect.

Access event log

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
apc>cd logs
E000: Success

apc>dir
E000: Success
           0 Jun 24 21:42  ./
           0 Jun 24 21:42  ../
       49120 Jun 24 11:24  event.txt
          13 Jun 24 21:52  data.txt
           1 Jun 24 21:42  dlogmask
      103424 Jun 24 11:11  ups_svc.txt

* Reboot required for change to take effect.

apc>eventlog
E000: Success


---- Event Log -------------------------------------------------

     Date: 06/25/2024       Time: 11:35:38
     -------------------------------------
     Smart-UPS 2200 : Online - Green Mode

     Date        Time      User    Event
     ---------------------------------------------------------------------------
     06/25/2024  11:35:28  apc     Web user 'apc' logged in from 192.168.152.1.
     06/25/2024  11:35:17  apc     SSH CLI user 'apc' logged in from
                                   192.168.152.1.
     06/25/2024  11:34:16  System  Event Log cleared.

Installing QuadStor Virtual Tape Library

Guestbook

Install pre-requisite

1
2
3
4
5
6
7
# yum -y install wget
# yum -y install gcc
# yum -y install httpd
# yum -y install perl
# yum -y install kernel-devel
# yum -y groupinstall "Server with GUI"
# yum -y install postgresql-server

Disable Linux firewall

1
# systemctl disable firewalld --now

Download and install QuadStor Virtual Tape Library

1
2
3
4
5
6
7
8
9
10
11
12
# wget -c https://www.quadstor.com/vtldownloads/quadstor-vtl-ext-3.0.79.14-rhel.x86_64.rpm
# rpm -Uivh quadstor-vtl-ext-3.0.79.14-rhel.x86_64.rpm
Preparing...                          ################################# [100%]
Adding group vtprocgrp
usermod -G vtprocgrp apache > /dev/null 2>&1
Updating / installing...
1:quadstor-vtl-ext-3.0.79.14-rhel  ################################# [100%]
Performing post install. Please wait...
Created symlink from /etc/systemd/system/multi-user.target.wants/quadstorvtl.service to /usr/lib/systemd/system/quadstorvtl.service.
Building required kernel modules
Running /quadstorvtl/bin/builditf. This may take a few minutes.
Installation complete. A system reboot is required to start the VTL service

Enable and Start Apache server

1
# systemctl enable httpd --now

Verify

Windows SSH: Permissions for 'private-key' are too open

Guestbook

Permissions for ‘id_rsa’ are too open.

1
2
3
4
5
6
7
8
9
C:\Users\sepmadm\.ssh>ssh -i id_rsa root@192.168.233.162
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions for 'id_rsa' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "id_rsa": bad permissions
root@192.168.233.162's password:

Fix

Synchronize Files With Unison On Linux

Guestbook

Download and install unison

1
2
3
4
5
6
7
8
9
10
11
# yum install -y --downloadonly epel-release.noarch
# rpm -Uivh /var/cache/dnf/extras-55926faaa9829325/packages/epel-release-9-5.el9.noarch.rpm 
# yum install -y --downloadonly unison
# yum install -y --downloadonly unison-doc.noarch

# cd /var/cache/dnf/epel-4b20c555de8aed94/packages/
# ls -l
-rw-r--r-- 1 root root 1453195 Jun 21 00:17 unison-2.53.5-1.el9.x86_64.rpm
-rw-r--r-- 1 root root  364206 Jun 21 00:17 unison-doc-2.53.5-1.el9.noarch.rpm

# rpm -Uivh unison-2.53.5-1.el9.x86_64.rpm unison-doc-2.53.5-1.el9.noarch.rpm

Version

1
2
# unison -version
unison version 2.53.5 (ocaml 4.11.1)

Sync file between local directories in one time

1
# unison /tmp/src/ /tmp/dst/

Sync file every 10s

1
# unison -repeat 10 -prefer newer /tmp/src/ /tmp/dst/

Problem Determination for AIX

Guestbook

sshd core dump

  1. AIX sshd got core dump

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    root@suuxinap01:/root# errpt -a
    ---------------------------------------------------------------------------
    LABEL:          CORE_DUMP
    IDENTIFIER:     A924A5FC

    Date/Time:       Tue Jun 18 00:27:00 HKT 2024
    Sequence Number: 5774
    Machine Id:      00C862804B00
    Node Id:         suuxinap01
    Class:           S
    Type:            PERM
    WPAR:            Global
    Resource Name:   SYSPROC

    Description
    SOFTWARE PROGRAM ABNORMALLY TERMINATED

    Probable Causes
    SOFTWARE PROGRAM

    User Causes
    USER GENERATED SIGNAL

            Recommended Actions
            CORRECT THEN RETRY

    Failure Causes
    SOFTWARE PROGRAM

            Recommended Actions
            RERUN THE APPLICATION PROGRAM
            IF PROBLEM PERSISTS THEN DO THE FOLLOWING
            CONTACT APPROPRIATE SERVICE REPRESENTATIVE

    Detail Data
    SIGNAL NUMBER
              11
    USER'S PROCESS ID:
                  20513234
    FILE SYSTEM SERIAL NUMBER
               1
    INODE NUMBER
                         2
    CORE FILE NAME
    //core
    PROGRAM NAME
    sshd
    CORE FILE NAME
    //core
    PROGRAM NAME
    sshd
    STACK EXECUTION DISABLED
               0
    COME FROM ADDRESS REGISTER
    rmgr_disa FFFFF9F4

    PROCESSOR ID
      hw_fru_id: 1
      hw_cpu_id: 12

    ADDITIONAL INFORMATION
    extend_br 2C4
    extend_br 27C

    Symptom Data
    REPORTABLE
    1
    INTERNAL ERROR
    0
    SYMPTOM CODE
    PCSS/SPI2 FLDS/sshd SIG/11 FLDS/extend_br VALU/2c4 FLDS/rmgr_disa
    ---------------------------------------------------------------------------

  2. Modify sshd_config

    1
    2
    root@suuxinap01:/etc/ssh# vi sshd_config
    ClientAliveInterval 180

  3. Restart sshd

    1
    2
    3
    root@suuxinap01:/etc/ssh# stopsrc -s sshd; startsrc -s sshd
    0513-044 The sshd Subsystem was requested to stop.
    0513-059 The sshd Subsystem has been started. Subsystem PID is 7602682.