2017-04-23

Recovering failed master database

As always, to recover a server, you need to have a backup!

USE master
GO
DUMP DATABASE master to '/home/dumps/master.dmp'
GO
SHUTDOWN

Here we additionaly have stopped the server to make it a little destroyed... Here you can check where the masterdevice exists and how to change its name:

[sap@SAPSybase install]$ ls
core.jsa_20161016_210528.1147  core.jsa_20170219_172003.1157  core.jsa_20170309_224222.1180  RUN_SAP01_BS       SAP01.log   veritas
core.jsa_20170219_121910.1154  core.jsa_20170219_173024.1170  core.jsa_20170327_022202.1180  SAP01_BS.log       showserver
core.jsa_20170219_170522.1147  core.jsa_20170219_180321.1181  RUN_SAP01                      SAP01_JSAGENT.log  sybhauser
[sap@SAPSybase install]$ more RUN_SAP01
#!/bin/sh
#
# SAP ASE page size (KB):       4k
# Master device path:   /opt/sap/data/master.dat
# Error log path:       /opt/sap/ASE-16_0/install/SAP01.log
# Configuration file path:      /opt/sap/ASE-16_0/SAP01.cfg
# Directory for shared memory files:    /opt/sap/ASE-16_0
# License properties file:      /opt/sap/ASE-16_0/sysam/SAP01.properties
# Interfaces file directory:    /opt/sap
# SAP Adaptive Server Enterprise name:  SAP01
#
/opt/sap/ASE-16_0/bin/dataserver \
-d/opt/sap/data/master.dat \
-e/opt/sap/ASE-16_0/install/SAP01.log \
-c/opt/sap/ASE-16_0/SAP01.cfg \
-M/opt/sap/ASE-16_0 \
-N/opt/sap/ASE-16_0/sysam/SAP01.properties \
-i/opt/sap \
-sSAP01 \

[sap@SAPSybase install]$ ls /opt/sap/data
empty  master.dat  sybmgmtdb.dat  sybsysdb.dat  sysprocs.dat  tempdbdev.dat
[sap@SAPSybase install]$ mv /opt/sap/data/master.dat /opt/sap/data/master.dat.org
[sap@SAPSybase install]$ ls /opt/sap/data
empty  master.dat.org  sybmgmtdb.dat  sybsysdb.dat  sysprocs.dat  tempdbdev.dat

Of course now Sybase cannot start:


[sap@SAPSybase install]$ ../bin/startserver -f ./RUN_SAP01
[sap@SAPSybase install]$ 00:0000:00000:00000:2017/04/23 19:54:39.86 kernel  Attempt to open device /opt/sap/data/master.dat failed.
00:0000:00000:00000:2017/04/23 19:54:39.86 kernel  dopen: open '/opt/sap/data/master.dat', No such file or directory

Next point is to start dataserver with options -d and -b to force it to create a new master device:

[sap@SAPSybase install]$ ../bin/dataserver -d /opt/sap/data/master.dat -b 100M
dataserver: Server name missing or is longer than supported. Proceeding with 'SYBASE' as Servername.
dataserver: master device size for this server: 100.0 Mb
dataserver: master database size for this server: 13.0 Mb
dataserver: model database size for this server: 3.0 Mb
00:0000:00000:00000:2017/04/23 19:58:04.57 kernel  SySAM: Using licenses from: /opt/sap/SYSAM-2_0/licenses/SYBASE_ASE_DE.lic
00:0000:00000:00000:2017/04/23 19:58:04.59 kernel  SySAM: Checked out license for 1 ASE_CORE (2018.1231/permanent/0534 6FD0 8CE5 427C).
00:0000:00000:00000:2017/04/23 19:58:04.59 kernel  This product is licensed to: ASE Developer Edition - For Development and Test use only
00:0000:00000:00000:2017/04/23 19:58:04.59 kernel  Checked out license ASE_CORE
00:0000:00000:00000:2017/04/23 19:58:04.59 kernel  Adaptive Server Enterprise (Developer Edition)
00:0000:00000:00000:2017/04/23 19:58:04.59 kernel  Performing space allocation for device '/opt/sap/data/master.dat' (0.10 Gb).  This may take some time.
00:0000:00000:00000:2017/04/23 19:58:04.59 kernel  Finished initialization.
....

At this point the master.dat device has been recreated, so we can start Sybase. Unfortunately this master database does not know anything about other databases previously created on the server. If this is not a problem for you - ok. But if you wish to have the configuration like before the failure, you need somehow to restore the master database. Let's begin with starting Sybase in single-user mode:

[sap@SAPSybase install]$ ../bin/startserver -f ./RUN_SAP01 -m

Now Sybase is started, but it is using it original configuration. There isn't any information about other databases, servers and even logins. The only login that is working is the "sa", and it's password is empty! Now connect to Sybase and define the location of backup server. This needs to be done manually, as the procedure sp_addserver is not working anymore! When updating the srvnetname use the name that can be fund in INTERFACES file:

[sap@SAPSybase install]$ isql -Usa -S SAP01
Password:
1> SELECT * FROM sysservers
2> go
 srvid  srvstatus srvname
         srvnetname                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
         srvclass srvsecmech
         srvcost srvstatus2
         srvprincipal                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
 ------ --------- ------------------------------------------------------------
      1         8 SYB_BACKUP
         SYB_BACKUP                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
             NULL NULL
            NULL          2
         NULL                                                                                                                                                                                                                                                                                                                                                                                                                                                                          

(1 row affected)
1> begin tran
2> update sysservers set srvnetname = 'SAP01_BS' where srvname='SYB_BACKUP'
3> commit
4> go
(1 row affected)
1> select * from sysservers
2> go
 srvid  srvstatus srvname
         srvnetname                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
         srvclass srvsecmech
         srvcost srvstatus2
         srvprincipal                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
 ------ --------- ------------------------------------------------------------
      1         8 SYB_BACKUP
         SAP01_BS                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
             NULL NULL
            NULL          2
         NULL                                                                                                                                                                                                                                                                                                                                                                                                                                                                          

(1 row affected)
1>

At this stage you should be ready to restore the master database:

1> load database master from '/home/dumps/master.dmp'
2> go
WARNING: In order to LOAD the master database, the ASE must run in single-user
mode. If the master database dump uses multiple volumes, you must execute
sp_volchanged on another ASE at LOAD time in order to signal volume changes.
Backup Server session id is: 64. Use this value when executing the
'sp_volchanged' system stored procedure after fulfilling any volume change
request from the Backup Server.
Backup Server: 6.28.1.1: Dumpfile name 'master1711311640 ' section number 1
mounted on disk file '/home/dumps/master.dmp'
Backup Server: 4.188.1.1: Database master: 4644 kilobytes (17%) LOADED.
Backup Server: 4.188.1.1: Database master: 26630 kilobytes (100%) LOADED.
Backup Server: 4.188.1.1: Database master: 26644 kilobytes (100%) LOADED.
Backup Server: 3.42.1.1: LOAD is complete (database master).
Started estimating recovery log boundaries for database 'master'.
Database 'master', checkpoint=(4749, 45), first=(4749, 45), last=(4750, 4).
Completed estimating recovery log boundaries for database 'master'.
Started ANALYSIS pass for database 'master'.
Completed ANALYSIS pass for database 'master'.
Started REDO pass for database 'master'. The total number of log records to
process is 8.
Redo pass of recovery has processed 1 committed and 0 aborted transactions.
Completed REDO pass for database 'master'.
Started filling free space info for database 'master'.
Completed filling free space info for database 'master'.
Started cleaning up the default data cache for database 'master'.
Completed cleaning up the default data cache for database 'master'.
(4 rows affected)
Database 'master' is now online.
CT-LIBRARY error:
        ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect
[sap@SAPSybase install]$ showserver
F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
0 S sap       1128  1118  0  80   0 -  6659 poll_s 10:02 ?        00:00:00 /opt/sap/ASE-16_0/bin/backupserver -e/opt/sap/ASE-16_0/install/SAP01_BS.log -N25 -C20 -I/opt/sap/interfaces -M/opt/sap/ASE-16_0/bin/sybmultbuf -SSAP01_BS

Why the command  ends with an error? That is because at the end of master database restore, the server shuts down and the connection is terminated.

The moment of the truth comes now. We need to start the server normally:

../bin/startserver -f ./RUN_SAP01

After that it should be possible to connect to the Sybase and work with as previously.

Possible problems with master restore:

If you would place  other databases on master, then they probably disappeared. If you have backups for them - you need to continue in a normal way. In my case the pubs2 and pubs3 databases were originally created on master device and now they were not working anymore.

If your master dump was not fresh enough, and in the meantime also new databases have been created, new logins added or new devices configured, you need to redo those actions manually. The biggest problem is with the devices, specially if on them already databases were created, that contain some data. The command "disk init" would overwrite the content of the devices, that's why we should use "disk reinit" command. (Be careful - incorrect options passed to the disk reinit, may corrupt the data!) This command adds entries in the devices table, but skips the initialization phase. After all the devices have been recreated, also sysdatabases needs to be rebuild. The command "disk refit" should be created for this. Disk refit stops the Sybase as it's last action.

Anther problem may be, if the master database was set up using non default settings. For example the page size could be set to 4K. If you followed previously shown steps, than following errors may be displayed during master database load:

1> load database master from '/home/dumps/master.dmp'
2> go
WARNING: In order to LOAD the master database, the ASE must run in single-user
mode. If the master database dump uses multiple volumes, you must execute
sp_volchanged on another ASE at LOAD time in order to signal volume changes.
Backup Server session id is: 60. Use this value when executing the
'sp_volchanged' system stored procedure after fulfilling any volume change
request from the Backup Server.
Backup Server: 4.177.2.1: The database page size of 2048 bytes obtained from ASE
is different from the database page size of 4096 bytes read from the dump
header. The LOAD session must exit.
Backup Server: 1.14.2.2: Unrecoverable I/O or volume error.  This DUMP or LOAD
session must exit.
Backup Server: 6.32.2.3: /home/dumps/master.dmp: volume not valid or not
requested (server: , session id: 60.)
Backup Server: 1.14.2.4: Unrecoverable I/O or volume error.  This DUMP or LOAD
session must exit.
Msg 8009, Level 16, State 1:
Line 1:
Error encountered by Backup Server.  Please refer to Backup Server messages for
details.

In this case you need to take a step back and create default empty master database build on 4K blocks. This can be achieved using the option -z of dataserver command:

[sap@SAPSybase install]$ isql -Usa -S SAP01
Password:
1>SHUTDOWN
2>GO

../bin/dataserver -d /opt/sap/data/master.dat -b 100M -z 4k
../bin/startserver -f ./RUN_SAP01 -m
[sap@SAPSybase install]$ isql -Usa -S SAP01
Password:
1>

Hope, this will never be useful for this, but in case it would, I hope to help ;)

1 comment:

SAP Adaprive Server Enterprise

SAP Adaprive Server Enterprise
SAP Adaprive Server Enterprise