Basic Steps to Manage Oracle ACFS Systems

To create and verify a file system, perform the following steps:

1. Create an Oracle ADVM volume in a mounted disk group with the ASMCMD volcreate command.

   The compatibility parameters COMPATIBLE.ASM and COMPATIBLE.ADVM must be set to 11.2 or higher for the disk group to contain an Oracle ADVM volume. To use Oracle ACFS encryption, replication, security, or tagging, the disk group on which the volume is created for the file system must have compatibility attributes for ASM and ADVM set to 11.2.0.2 or higher.

   Start ASMCMD connected to the Oracle ASM instance. You must be a user in the OSASM operating system group.

   When configuring Oracle ADVM volume devices within a disk group, Oracle recommends assigning the Oracle Grid Infrastructure user and Oracle ASM administrator roles to users who have root privileges.

   To create a volume:

   ASMCMD [+] > volcreate -G data -s 10G volume1
   

   When creating an Oracle ADVM volume, a volume device name is created that includes a unique Oracle ADVM persistent disk group number. The volume device file functions in the same manner as any other disk or logical volume to mount file systems or for applications to use directly.

   The format of the volume name is platform-specific.

2. Determine the device name of the volume that was created.

   You can determine the volume device name with the ASMCMD volinfo command or from the VOLUME_DEVICE column in the V$ASM_VOLUME view.

   For example:

   ASMCMD [+] > volinfo -G data volume1
   Diskgroup Name: DATA
   
            Volume Name: VOLUME1
            Volume Device: /dev/asm/volume1-123
            State: ENABLED
        ... 
   
   SQL> SELECT volume_name, volume_device FROM V$ASM_VOLUME 
        WHERE volume_name ='VOLUME1';
   
   VOLUME_NAME        VOLUME_DEVICE
   -----------------  --------------------------------------
   VOLUME1            /dev/asm/volume1-123
   

3. Create a file system with the Oracle ACFS mkfs command.

   Create a file system using an existing volume device.

   For example:

   $ /sbin/mkfs -t acfs /dev/asm/volume1-123
   
   mkfs.acfs: version                   = 11.2.0.1.0.0
   mkfs.acfs: on-disk version           = 39.0
   mkfs.acfs: volume                    = /dev/asm/volume1-123
   mkfs.acfs: volume size               = 10737418240
   mkfs.acfs: Format complete.
   

   The root privilege is not required to run mkfs. The ownership of the volume device file dictates who can run this command.

4. Optionally register the file system with the acfsutil registry command.

   For example:

   $ /sbin/acfsutil registry -a /dev/asm/volume1-123 /acfsmounts/acfs1
   
   acfsutil registry: mount point /acfsmounts/acfs1 successfully added 
     to Oracle Registry
   

   The root or asmadmin privileges are required to modify the registry. The Windows Administrator privilege is equivalent to the root privilege on Linux.

   Registering a file system is optional. After registering an Oracle ACFS file system in the cluster mount registry, the file system is mounted automatically on each cluster member listed in the registry entry during the next registry check action. This automatic process runs every 30 seconds and eliminates the requirement to manually mount the file system on each member of the cluster.

   Registering an Oracle ACFS file system also causes the file system to be mounted automatically whenever Oracle Clusterware or the system is restarted.

   Note:

   In an Oracle Grid Infrastructure Clusterware configuration, you can run srvctl add filesystem to automount a file system; this method is required when an Oracle Database home is installed on an Oracle ACFS file system. However, that file system should not be added to the registry.

   Note:

   Oracle ACFS registration (acfsutil registry) is not supported in an Oracle Restart (standalone) configuration, which is a single-instance (non-clustered) environment.

5. Mount the file system with the Oracle ACFS mount command. You can mount a file system before or after registering the file system. If the file system has been registered, you can wait for the file system to be mounted automatically.

   For example:

   # /bin/mount -t acfs /dev/asm/volume1-123 /acfsmounts/acfs1
   

   The root privilege is required to run the mount command and the Windows Administrator privilege is required to run the acfsmountvol command.

   After the file system has been mounted, ensure that the permissions are set to allow access to the file system for the appropriate users. For example:

   # chown -R oracle:dba /acfsmounts/acfs1
   

6. Create a test file in the file system.

   The user that creates the test file should be a user that is intended to access the file system. This test ensures that the appropriate user can write to the file system.

   For example:

   $ echo "Oracle ACFS File System" > /acfsmounts/acfs1/myfile
   

7. List the contents of the test file that was created in the file system.

   For example:

   $ cat /acfsmounts/acfs1/myfile
   Oracle ACFS File System
   

If the node is part of a cluster, perform the following steps on node 2 to view the test file you created on node 1.

1. Enable the volume that was previously created and enabled on node 1.

   Start ASMCMD connected to the Oracle ASM instance. You must be a user in the OSASM operating system group.

   For example:

   ASMCMD [+] > volenable -G data volume1
   

2. View information about the volume that you created on node 1.

   For example:

   ASMCMD [+] > volinfo -G data volume1
   

3. Mount the file system using the Oracle ACFS mount command.

   For example:

   # /bin/mount -t acfs /dev/asm/volume1-123 /acfsmounts/acfs1
   

   The root privilege is required run the mount command and the Windows Administrator privilege is required to run the acfsmountvol command.

   After the file system has been mounted, ensure that the permissions are set to allow access for the appropriate users.

4. List the contents of the test file you previously created on the file system.

   For example:

   $ cat /acfsmounts/acfs1/myfile
   Oracle ACFS File System
   

   The contents should match the file created previously on node 1.

To create and verify a snapshot on node 1:

The disk groups on which volumes are created for the primary and standby file systems must have compatibility attributes for ASM and ADVM set to 12.2 or higher.

The steps to manage replication are:

1. Ensure that ssh has been configured for replication.

   • Ensure that host keys and user keys for the ssh command have been configured on your primary and standby clusters.

   • On Windows, ensure that Cygwin is installed and ssh is configured as required for replication.

2. Ensure that the snapshots needed by replication can be created at all times. At any given point replication, may need to be able to use two concurrent snapshots of the primary file system, and one snapshot of the standby file system.

3. Ensure that there is adequate network connectivity between the primary and standby sites. You should verify that the achievable network data transfer rate from primary to standby is substantially larger than the rate of change of data on the primary file system.

   One way to estimate network data transfer rate is to start with an observed transfer rate, then reduce it to account for known sources of overhead. For example, you can calculate the elapsed time needed to FTP a 1 G file from the primary file system to the intended standby file system, during a period when network usage is low. This provides an estimate of the maximum achievable transfer rate. This rate should be reduced to account for overheads inherent in replication transfers, as well as to allow for other demands on the network. For replication overhead, a reasonable approach is to reduce the measured rate by 20%, then by an additional 5% for each node in the primary cluster.

   To estimate the average rate of change on the primary, you can use the command acfsutil info fs with the -s option. This command should be run on each node where the primary file system is mounted, and displays that amount and rate of change to the file system on that node. To compute the total rate of change for the file system, the rate of changed for each node must be aggregated. A reasonable value to use for -s is 900, which would yield a 15 minute sampling interval.

   With the output from acfsutil info fs with the -s option, you can determine the average rate of change, the peak rate of change, and how long the peaks last. A conservative approach to using this data is to choose the peak rate of change as the target rate that must be accommodated.

   Because replication must transfer all data changed on the primary to the standby, obviously the achievable network transfer rate must be higher, ideally significantly higher, than the target rate of change on the primary. If this is not the case, you should increase network capacity before implementing replication for this file system and workload.

   For example, assume you have a four node primary cluster and you determine that a 1 G file can be transferred in 30 seconds, yielding a current FTP transfer rate of 33 M per second. An estimate of the current replication transfer rate would be approximately 20 M per second, calculated as follows:

   33 MB/sec * (1 – 0.2 – (4 * 0.05)) = 33 * 0.6 = ~20 MB/sec
   

   Also, you find that the average rate of change to the primary is 8 GB per hour, with a peak rate of 25 G per hour. Using the peak rate, you can calculate a target rate of change of approximately 7 M per second as follows:

   (25 GB/hour * 1024) / 3600 = ~7 MB/sec
   

   In the scenario that was discussed in this step, you can reasonably expect the network to be able to handle the additional workload from replication.

4. Ensure that there is adequate storage capacity on the primary and standby sites.

   Estimate the storage capacity needed for replication on the sites hosting the primary and standby file systems. In the general case, the primary site must store two snapshots of the primary file system on an ongoing basis and the standby site must store a single snapshot of the standby file system. The space occupied by these snapshots mostly consists of user data or metadata preserved in the snapshot, that has since been modified which triggers a new copy of the data to be created.

   The space occupied by replication-related snapshots can be directly viewed using the command acfsutil snap info. On the primary, check for snapshots with the names starting with the string REPL. On the standby, look for snapshots for names starting with SDBACKUP.

   If you use interval-based replication, the -i option to acfsutil repl init primary, and if the replication operations are successfully completing within the specified interval, then the size of replication-related snapshots is related to the rate of change of the primary and the length of the interval. For example, with an average rate of change of 8 G per hour and a two hour replication interval, you would expect that snapshot storage usage is in the range of 16 G per snapshot.

   Snapshot size does vary with the rate of change of the primary. Another factor is that snapshot size depends in part on the number of files in the file system, as well as the rate of change. Potentially more importantly, if you use constant mode replication, the -C option to acfsutil repl init primary, or if replication operations are not completing successfully in the interval given with interval—based replication because the interval is too small, the size of replication-related snapshots is difficult to predict in advance. In these cases, observe the size of the snapshots being generated over time and adjust the file system size as needed with the acfsutil size command to accommodate normal storage needs in addition in the presence of the snapshots. When collecting this information, a good starting point is to accommodate space for the snapshots to contain the data that is multiple times larger than the collection period, at the average rate of change of the primary.

   While collecting this information, choose a conservative starting point for the amount of space to allow for replication snapshots. For example, you can compute the space needed to store changes to the file system over the collection period as described previously, then you can allocate several times that space for future snapshots.

5. Determine the user to be employed for replication and optionally set up tags.

   Choose or create the replication user who logs in with ssh to the standby cluster to apply data replicated from the primary file system to the standby file system. This user is defined only at the OS level and not within Oracle. The user should belong to the groups defined for Oracle ASM administrator access.

   Optionally set tags on directories and files to replicate only selected files in an Oracle ACFS file system. You can also add tags to files after replication has started.

6. Configure the site hosting the standby file system.

   Before replicating an Oracle ACFS file system, configure the site hosting the standby file system by performing the following:

   • Create a new standby file system of adequate size to hold the files replicated from the primary file system, as well as a single replication snapshot . For example:

     /standby/repl_data

   • Mount the file system on one node only.

   • Run the acfsutil repl init standby command on the site hosting the standby file system. For example:

     # /sbin/acfsutil repl init standby -u repluser /standby/repl_data
     

     Note:

     If the acfsutil repl init standby command is interrupted for any reason, the user must re-create the file system, mount it on one node only, and re-run the command.

     This command requires the name of the replication user and the standby file system. The specified user is the user under which ssh, invoked from the primary cluster, logs in to the standby cluster to apply changes. This user is specified with the -u option. For example: -u repluser.

     The mount point is the standby file system. For example: /standby/repl_data.

   • After the acfsutil repl init standby command has successfully completed, mount the specified file system on all nodes of the standby cluster.

   The acfsutil repl init standby command requires root or system administrator privileges to run.

7. After the standby file system has been set up, configure the site hosting the primary file system and start replication.

   Run the acfsutil repl init primary command on the site hosting the primary file system. For example:

   $ /sbin/acfsutil repl init primary -i 2h -s repluser@standby12_vip -m /standby/repl_data /acfsmounts/repl_data
   

   This command requires the following configuration information.

   • A replication interval, given with the option -i interval for interval mode or the option -C for constant mode replication. If an interval is specified, the option value is the minimum amount of time that elapses between replication operations. At the start of each operation, replication takes a new snapshot of the primary and compares it to the previous snapshot, if any. The changes needed to update the standby to match the primary are then sent to the standby. If -C is given instead of -i interval, a new replication operation is started as soon as the previous one completes.

     For example, to set up a replication interval of two hours, specify -i 2h.

   • The user name and network endpoint (VIP name or address, or host name or address) to be used to connect to the site hosting the standby file system, specified with the —s option. For example: -s repluser@standby12_vip

   • The mount point of the primary file system. For example: /acfsmounts/repl_data

   • If the mount point is different on the site hosting the standby file system than it is on the site hosting the primary file system, specify the mount point on the standby file system with the -m standby_mount_point option. For example: -m /standby/repl_data

   The acfsutil repl init primary command requires root or system administrator privileges to run.

8. Monitor information about replication on the file system.

   The acfsutil repl info command displays information about the state of the replication processing on the primary or standby file system.

   For example, you can run the following on the site hosting the primary file system to display configuration information.

   $ /sbin/acfsutil repl info -c -v /acfsmounts/repl_data
   

   You must have system administrator (the user root on non-Windows systems or local SYSTEM on Windows) or Oracle ASM administrator privileges to run this command.

9. Manage the replication background process.

   Run the acfsutil repl bg command to start, stop, or retrieve information about the replication background process.

   For example, run the following command to display information about the replication process for the /acfsmounts/repl_data file system.

   $ /sbin/acfsutil repl bg info /acfsmounts/repl_data
   

   You must have system administrator or Oracle ASM administrator privileges to run the acfsutil repl bg info command.

10. Pause replication momentarily if necessary.

    Run the acfsutil repl pause to momentarily stop replication. Run the acfsutil repl resume command as soon as possible to resume replication.

    For example, the following command pauses replication on the /acfsmounts/repl_data file system.

    $ /sbin/acfsutil repl pause /acfsmounts/repl_data
    

    The following command resumes replication on the /acfsmounts/repl_data file system.

    $ /sbin/acfsutil repl resume /acfsmounts/repl_data
    

    You must have system administrator or Oracle ASM administrator privileges to run the acfsutil repl pause and acfsutil repl resume commands.

11. Failing over to a standby or turning a standby file system into an active file system.

    If the primary file system is inaccessible, you can run acfsutil repl terminate standby mount_point to turn the standby file system into an active file system. If the primary file system still exists, you should terminate the primary first with acfsutil repl terminate primary mount_point.

    Before terminating replication with acfsutil repl terminate standby on the standby file system, you can determine the point in time of the primary file system that the standby file system represents. This timestamp is displayed with acfsutil repl info -c as Last sync time with primary. If the failover action must be coordinated with Oracle Data Guard, you can use the timestamp to set back the database if needed, or perform other necessary actions that are based on the timestamp.

    The standby file system may be in the process of being modified by replication. This could occur if:

    • The primary file system is available and a replication operation is currently in progress.

    • The primary file system is not available, but a replication operation was in progress when it became unavailable.

    To be sure of obtaining the contents of the standby file system when it was last identical to a snapshot from the primary, follow one of these procedures.

    • If the primary file system is available, run the acfsutil repl terminate primary command on the primary site to terminate replication. The command waits for any in-progress replication operation to complete before it returns. Then run acfsutil repl info -c to determine the point in time of the primary file system represented on the standby. After you have this information, run acfsutil repl terminate standby on the standby site.

    • If the primary file system is not available, you should first compare two date strings from the acfsutil repl info -c output; these are the dates from the Receiving primary as of line and the Last sync time with primary line. If these dates are identical, then the standby file system contains the most recent available point-in-time image of the primary. If they are not equal, you must use the backup snapshot recorded by replication to recover the last point-in-time image captured on the standby. You can find this snapshot using the acfsutil snap info command. Search for a snapshot with a name of the form:

      SDBACKUP_tstamp1_REPL_tstamp2
      

      where tstamp1 represents the time at which the backup snapshot was created, and tstamp2 represents the point in time when the primary contents in this snapshot were recorded. There should be only one backup snapshot present. The date of the backup snapshot corresponds to the date in the Last sync time with primary line output by acfsutil repl info -c. If there is no backup snapshot available, then the contents of the primary were never successfully transferred to the standby.

      The date of the backup snapshot corresponds to the date in the Last sync time with primary line output by acfsutil repl info -c.

      To use the backup snapshot, you must terminate replication and ensure that the snapshot is preserved because it is deleted by default. To ensure the snapshot is preserved, add the -k option to the command line for acfsutil repl terminate standby. After replication has been terminated, you can run the acfsutil snap remaster command to use the snapshot as the new contents of the standby file system.

To permanently remove a volume and Oracle ACFS file system, perform the following steps. These steps destroy the data in the file system.

1. Deregister the file system with acfsutil registry -d.

   For example:

   $ /sbin/acfsutil registry -d /acfsmounts/acfs1
   acfsutil registry: successfully removed ACFS mount point
      /acfsmounts/acfs1 from Oracle Registry
   

2. Dismount the file system.

   For example:

   # /bin/umount /acfsmounts/acfs1
   

   You must dismount the file system on all nodes of a cluster.

   Use umount on Linux systems or acfsdismount on Windows systems.

3. Remove the file system with acfsutil rmfs.

   If you were not planning to remove the volume in a later step, this step is necessary to remove the file system. Otherwise, the file system is removed when the volume is deleted.

   For example:

   $ /sbin/acfsutil rmfs /dev/asm/volume1-123
   

4. Optionally you can disable the volume with the ASMCMD voldisable command.

   For example:

   ASMCMD> voldisable -G data volume1
   

5. Delete the volume with the ASMCMD voldelete command.

   For example:

   ASMCMD> voldelete -G data volume1