Tuesday, December 22, 2015

File - Object Unified Access

What is unified-access ?

Unified file and object access allows use cases where you can access data using object as well as file interfaces. For example: If a user ingests a file from the SMB interface then users with valid access rights can access that file from the object interface. On the other hand, if a user ingests a object from object interface then users with valid access rights can access that file from file interface.

Why this post ?

  1. Configuration of Spectrum Scale for unified access
  2. Demo of unified access.
Prerequisite: Spectrum Scale 4.2+ should be installed.

Details of cluster which I'll be using for demo::

[root@vwnode4 ~]# mmlscluster

GPFS cluster information
  GPFS cluster name:         vwnode.gpfscluster
  GPFS cluster id:           XXXX548474453088585
  GPFS UID domain:           vwnode.gpfscluster
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp
  Repository type:           CCR

 Node  Daemon node name  IP address    Admin node name  Designation
   1   vwnode0           XX.XX.100.110  vwnode0          quorum-perfmon
   2   vwnode1           XX.XX.100.111  vwnode1          quorum-perfmon
   3   vwnode2           XX.XX.100.112  vwnode2          quorum-perfmon
   4   vwnode3           XX.XX.100.113  vwnode3          manager-perfmon
   5   vwnode4           XX.XX.100.114  vwnode4          manager-perfmon

User authentication details

[root@vwnode4 ~]# mmuserauth service list
FILE access configuration : LDAP
ENABLE_SERVER_TLS        false
ENABLE_KERBEROS          false
USER_NAME                cn=manager,dc=example,dc=com
SERVERS                  XX.XX.46.17
NETBIOS_NAME             st001
BASE_DN                  dc=example,dc=com
USER_DN                  none
GROUP_DN                 none
NETGROUP_DN              none
USER_OBJECTCLASS         posixAccount
GROUP_OBJECTCLASS        posixGroup
USER_ID_ATTRIB           uid
KERBEROS_SERVER          none
KERBEROS_REALM           none

OBJECT access configuration : LDAP
ENABLE_SERVER_TLS        false
ENABLE_KS_SSL            false
USER_NAME                cn=manager,dc=example,dc=com
SERVERS                  XX.XX.46.17
BASE_DN                  dc=example,dc=com
USER_DN                  ou=people,dc=example,dc=com
USER_OBJECTCLASS         posixAccount
USER_ID_ATTRIB           uid
USER_MAIL_ATTRIB         mail
USER_FILTER              none
KS_ADMIN_USER            ldapuser3

Configuration of Unified Access

Step 1: Enable the file-access object capability from any protocol node

[root@vwnode4 ~]# mmobj config change --ccrfile spectrum-scale-object.conf --section capabilities --property file-access-enabled --value true

To validate whether unified access is enable you can check status ibmobjectizer service.
If unified access is enabled ibmobjectizer must be running on exactly one node.

[root@vwnode4 ~]# mmces service list -v --all | grep ibmobjectizer
vwnode3:        OBJ:ibmobjectizer                            is running

Step 2: For this demo, I am using unified_mode for authentication.
In unified_mode users from object and file are expected to be common and coming from the same directory service (Note that I have LDAP user authentication configure for both object and file).
Check this for more information.

[root@vwnode4 ~]# mmobj config change --ccrfile object-server-sof.conf --section DEFAULT --property id_mgmt --value unified_mode

Step3: Create policy for unified access.
Following command will create policy with name 'swiftOnFile' with unified access enabled.

[root@vwnode4 ~]# mmobj policy create swiftOnFile --enable-file-access
[I] Getting latest configuration from ccr
[I] Creating fileset /dev/cesSharedRoot:obj_swiftOnFile
[I] Creating new unique index and building the object rings
[I] Updating the configuration
[I] Uploading the changed configuration

Let's check our freshly created policy for unified access.

[root@vwnode4 ~]# mmobj policy list

Index       Name         Default Deprecated Fileset           Functions
0           SwiftDefault yes                my_object_fileset
56921512210 swiftOnFile                     obj_swiftOnFile   file-and-object-access

You can make this policy default, though it is optional.

[root@vwnode4 ~]# mmobj policy change swiftOnFile --default

Demo of Unified Access

Now let's create a container and add a file in it.
I am going to use Swift Explorer for this.
If you are new to Swift Explorer please check my previous post to configure Swift Explorer -
Accessing Spectrum Scale Object Store using Swift Explorer

Create a container :


Upload a file :


Let's check where this file is uploaded on server.

[root@vwnode4 ~]# ls -l /ibm/cesSharedRoot/obj_swiftOnFile/s56921512210z1device1/AUTH_2de13f0dae4747b484ed06bc31b29835/unified_access
total 0
-rwxr-xr-x. 1 ldapuser3 ldapuser3 11 Dec 21 09:37 file1.txt

Explanation for path :

/ibm/cesSharedRoot      -- Mount point for GPFS file system
obj_swiftOnFile         -- Policy create CLI creates a directory depending upon your policy name
s56921512210z1device1   -- 's' followed by policy index followed by fixed suffix 'z1device1'  
AUTH_2de13f0dae4747b484ed06bc31b29835 -- Unique ID for a tenet with fixed prefix 'AUTH_'
unified_access          -- Name of container
Let's export this container with NFS check this file from file interface.

[root@vwnode4 ~]# mmnfs export add /ibm/cesSharedRoot/obj_swiftOnFile/s56921512210z1device1/AUTH_2de13f0dae4747b484ed06bc31b29835/unified_access/ -c "*(Access_Type=RW,SecType=sys,Squash=NoIdSquash,Protocols=3:4)"
[root@vwnode4 ~]# mmnfs export list
Path                                                                                                          Delegations Clients
/ibm/cesSharedRoot/obj_swiftOnFile/s56921512210z1device1/AUTH_2de13f0dae4747b484ed06bc31b29835/unified_access none        * 

Let mount it on some other machine --

[root@localhost ~]# mount -t nfs -o vers=3 viknode:/ibm/cesSharedRoot/obj_swiftOnFile/s56921512210z1device1/AUTH_2de13f0dae4747b484ed06bc31b29835/unified_access /mnt/

Let's check our 'file1.txt'

[root@localhost ~]# ls -ln /mnt/
total 0
-rwxr-xr-x. 1 1002 1002 29 Dec 22 11:41 file1.txt
[root@localhost ~]# id ldapuser3
uid=1002(ldapuser3) gid=1002(ldapuser3) groups=1002(ldapuser3)

Now try to ingest a file from NFS and try to get it from object interface

[root@localhost ~]# su ldapuser3
bash-4.2$ echo "NFS Create File" > /mnt/nfs_file.txt
bash-4.2$ ls /mnt/
file1.txt  nfs_file.txt

Let's check this new file from object interface.

You can get more information about unified access here.

Upgrading from 4.1.1 to 4.2

Why this post ?

Upgrading spectrum scale is as easy as pie. In this post I demonstrate how to upgrade Spectrum Scale from 4.1.1 to 4.2.0.

Where to start ?

Let's start with a Spectrum Scale 4.1.1 cluster.

Here is my 4.1.1 Cluster details -

[root@vwnode1 ~]# mmlscluster

GPFS cluster information
  GPFS cluster name:         vwnode.gpfscluster
  GPFS cluster id:           1175018713363073732
  GPFS UID domain:           vwnode.gpfscluster
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp
  Repository type:           CCR

 Node  Daemon node name  IP address    Admin node name  Designation
   1   vwnode1           XXX.XXX.100.111  vwnode1          quorum
   2   vwnode2           XXX.XXX.100.112  vwnode2          quorum
   3   vwnode3           XXX.XXX.100.113  vwnode3          quorum-manager
   4   vwnode4           XXX.XXX.100.114  vwnode4          manager

Let's upgrade

Extract brand new Spectrum Scale ( in my case )

[root@vwnode1 ~]# ./Spectrum_Scale_install- --silent > /dev/null

You can either fill-up all cluster definition using 'specturmscale' command or just copy old cluster definition file if there is no change in cluster since installation.

[root@vwnode1 images]# cp 
cp: overwrite â/usr/lpp/mmfs/â? y
[root@vwnode1 images]# 

 You will see lots of warnings if you check for object parameters

[root@vwnode1 images]# /usr/lpp/mmfs/ config object
[ INFO  ] No changes made. Current settings are as follows:
[ INFO  ] Object File System Name is cesSharedRoot
[ INFO  ] Object File System Mountpoint is /ibm/cesSharedRoot
[ INFO  ] Endpoint Hostname is viknode
[ WARN  ] No value for GPFS Object Base in clusterdefinition file.
[ WARN  ] No value for GPFS Fileset inode allocation in clusterdefinition file.
[ WARN  ] No value for Admin Token in clusterdefinition file.
[ WARN  ] No value for Admin User in clusterdefinition file.
[ WARN  ] No value for Admin Password in clusterdefinition file.
[ WARN  ] No value for Swift User in clusterdefinition file.
[ WARN  ] No value for Swift Password in clusterdefinition file.
[ WARN  ] No value for Database Password in clusterdefinition file.
[ INFO  ] Enable S3 is off
[ WARN  ] No value for Multi-region Data File Path in clusterdefinition file.
[ WARN  ] No value for Region Number for Multi-region in clusterdefinition file.

We need to specify following parameters in order to configure upgrade.

ap  -- Specify the password for the Admin User
dp  -- Specify the password for object database
i     -- Specify the GPFS fileset inode allocation to be used
o    -- Specify the GPFS fileset to be created/used as the

[root@vwnode1 ~]# /usr/lpp/mmfs/ config object -o my_object_fileset -ap Passw0rd -dp Passw0rd -i 10000
[ INFO  ] Setting GPFS Object Base to my_object_fileset
[ INFO  ] Setting GPFS Fileset inode allocation to 10000
Enter the secret encryption key:
Repeat the secret encryption key:
[ INFO  ] Setting Admin Password
[ INFO  ] Setting Database Password

We also need to configure perform monitoring.
You can either enable or disable performance monitoring as per your requirement.

[root@vwnode1 ~]# /usr/lpp/mmfs/ config perfmon -r on
[ INFO  ] Setting Enable Performance Monitoring reconfiguration to on

We should do precheck before upgrade.

[root@vwnode1 ~]# /usr/lpp/mmfs/ upgrade -pr

If upgrade pre-check is successful,  we are ready for upgrade.
Just fire following command which will do upgrade -

[root@vwnode1 ~]# /usr/lpp/mmfs/ upgrade

Done !

Let's confirm using 'mmfsadm'

[root@vwnode4 ~]# mmfsadm dump version | head -1
Build branch " ".