Recovering From SAP Mobile Server Failures

Add application code to check for and recover from from SAP Mobile Server failures.

It is highly recommended that you add a catch call to all synchronize methods (synchronize(), begingSynchronize(), and so on) within your applications to allow the application to recover if SAP Mobile Server fails and needs to be restored from an older database. If not, you may have to reinstall the application manually for all users so they can resynchronize with SAP Mobile Server.

See Restoring from an Older Backup Database File (Data Loss) in the System Administration Guide for information about SAP Mobile Server recovery.

As a best practice, and not included in these examples, application developers should include code that informs mobile application users about:
  • What is going to happen (for example, reregistering, recreating the local database, and so on). And,
  • The reason for the action (for example, lost registration, server is restored, and so on).
And prompt them for confirmation before executing the code.
When the SAP Mobile Server is restored to a previous state, it may be inconsistent with the state of the client. For example:
  • The client synchronizes with the server after the database is backed up. In this case the client cannot synchronize successfully with the server after the database is restored.
  • The client registers with the server after the database is backed up. In this case the client registration is lost when the database was restored.
The following sample code illustrates how the client can recover from these errors.
  1. After SAP Mobile Server is restored, client application connection information may be lost if the registration was created after the database was backed up. This client application calls startConnection to connect to the server, the onConnectionStatusChanged callback returns error code 580 with a message that authentication failed. The user can reregister the application with the ApplicationCallback implementation after encountering this error code. If the server is restored to a point-in-time when the client application has registered, the application runs as normal without receiving this error code. These examples illustrate both automatic and manual registration recovery.
    1. Automatic registration recovery:
      public void startApplication( )
        {
            Application app = Application.getInstance();
            Application.getInstance().setApplicationCallback(new MyApplicationCallback());
            try
            {
                 ConnectionProperties connProperties = app.getConnectionProperties();
                 connProperties.setServerName (“mega-vm008” );
                 connProperties.setPortNumber (5001);
                 connProperties.setLoginCredentials(new com.sybase.persistence.LoginCredentials(“test@admin”, “test123”));
                 if (app.getRegistrationStatus() == RegistrationStatus.UNREGISTERED)
                 {
                     app.registerApplication(100); // or call app.registerApplication();
                 }
                 else
                 {
                     app.startConnection(100); // or call app.startConnection();
                 }
            }
            catch (ApplicationRuntimeException ex)
            {
                System.out.println(ex);
            }
            catch (ApplicationTimeoutException ex)
            {
                System.out.println(ex);
            }
            while(app.getConnectionStatus() != ConnectionSatus.CONNECTED || app.getRegistrationStatus() != 
                      RegistrationStatus.REGISTED)
            {
                  try
                 {
                      Thread.sleep(100);
                 }
                 catch (Exception ex)
                {
                     System.out.println(ex);
                }
      
             }
      
        }
       public class MyApplicationCallback  extends  com.sybase.mobile.DefaultApplicationCallback
        {
            boolean callFlag = false;
           //override this method to check if need to reregister
            public  void onConnectionStatusChanged(int connectionStatus, int errorCode, String errorMessage)
            {
                if (errorCode == 580 && !callFlag)
                {
                   //this callback will be invoked multiple times when this error occures, but we just call once reregister, so set the
                   //callFlag to be true. 
                    callFlag = true;
                    Thread registerThread = new Thread(“reregister”)
                     {
                          public void run()
                         {
                             //do not unregister application, because the application connection info has been deleted from server side. We can 
                             //call register application directly.
                            Application.getInstance().registerApplication();
                         }
                     }.start();
                }
            }
      
        }
    2. Manual registration recovery. If error code 580 is encountered, the Administrator must first manually register the application in SAP Control Center, or else the reregistered application fails the first time. Manual registration requires setting of the activation code:
      public void startApplication( )
      {
          Application app = Application.getInstance();
          Application.getInstance().setApplicationCallback (new MyApplicationCallback());
      
          try
          {
               ConnectionProperties connProperties = app.getConnectionProperties();
               connProperties.setServerName(“mega-vm008” );
               connProperties.setPortNumber (5001);
               connProperties.setActivationCode (“100” );
               connProperties.setLoginCredentials (new com.sybase.persistence.LoginCredentials(“test@admin”, null));
               if (app.getRegistrationStatus() == RegistrationStatus.UNREGISTERED)
               {
                   app.registerApplication(100); // or call app.registerApplication();
               }
               else
               {
                   app.startConnection(100); // or call app.startConnection();
               }
          }
          catch (ApplicationRuntimeException ex)
          {
              System.out.println(ex);
          }
          catch (ApplicationTimeoutException ex)
          {
              System.out.println(ex);
          }
          while(app.getConnectionStatus() != ConnectionSatus.CONNECTED || app.getRegistrationStatus() != 
                    RegistrationStatus.REGISTED)
          {
               try
               {
                    Thread.sleep(100);
               }
               catch (Exception ex)
               {
                   System.out.println(ex);
              }
          }
      
        }
      public class MyApplicationCallback  extends  com.sybase.mobile.DefaultApplicationCallback
      {
          boolean callFlag = false;
         //override this method to check if need to reregister
          public  void onConnectionStatusChanged(int connectionStatus, int errorCode, String errorMessage)
          {
              if (errorCode == 580 && !callFlag)
              {
                 //this callback will be invoked multiple times when this error occures, but we just call once reregister, so set the
                 //callFlag to be true. 
                  callFlag = true;
                  Thread registerThread = new Thread(“reregister”)
                   {
                        public void run()
                        {
                           //do not unregister application, because the application connection info has been deleted from server side. We can 
                           //call register application directly.
                           Application.getInstance().registerApplication();
                        }
                    }.start();
               }
           }
       }
  2. Client Application RBS synchronization recovery.
    If the server is restored to a point-in-time of an application's previous synchronization, the client synchronization gets com.sybase.persistence.SynchronizeException with an error code of SQLE_UPLOAD_FAILED_AT_SERVER. This error code indicates the need to recover the client database. If the server is restored to a point-in-time of the application's last synchronization or the application has never synchronized, the client application can synchronize as normal without the exception. For example:
    • Time1: application registered and has not synchronized.
    • Time2: application synchronized for the first time.
    • Time3: application synchronized for the second time.
    • Time4: application synchronized for the last time.
    If the server is restored to time1 or time4, the client can synchronize successfully. If the server is restored to time2 or time3, client synchronization fails with com.sybase.persistence.SynchronizeException. You have three methods to recover the client database and synchronize successfully again:
    1. Before synchronization recovery, the application needs to complete application registration recovery if necessary.
    2. Once the client application starts, the application checks if the last recovery failed in the middle by checking the saved flag. If the last recovery failed, the application needs to resume the recovery first.
    3. Mark the recovery state. For example, the application can save the recovery state to a file. In recovery method two, the old client database is copied and used as a recovery state flag.
    1. Recreate database without copying old data (all data lost)
      This is the simplest recovery method, but the old data, such as synchronization parameters, SIS, and Local MBOs are not copied to the new database. The application user needs to reenter them in the application's GUI.
      try
       {
            End2end_rdbDB.synchronize();
       }
       catch (Exception ex)
       {
           //if meet this error, the server has been restored, we need to recover client database
             if (ex instanceof com.sybase.persistence.SynchronizeException)
             {
                if (((com.sybase.persistence.SynchronizeException) ex).getErrorCode() ==  
                       com.sybase.persistence.SynchronizeException.SQLE_UPLOAD_FAILED_AT_SERVER)
                {
                     recoverClientDatabase();
                }
             }
          }
      
          private void recoverClientDatabase()
          {
              setRecoveringInPlaceFlag();  //Like save a flag into FileSystem
              End2end_rdbDB.closeDBConnection();
              End2end_rdbDB.deleteDatabase();
              cleanRecoveringInPlaceFlag();
          }
    2. Recreate database and copying old database (local transaction lost)
      Copies old database data to a new database; this example includes personalization keys, subscription information, SIS info, local BO. Unsubmitted transactions like an MBO’s pending state are lost. This sample code checks if a copy of the database is available to determine if a recovery was interrupted.
      if(isRecoverFailed())
      {
           recoverClientDatabase();
      }
      else
      {
           try
           {
              End2end_rdbDB.synchronize();
           }
           catch (Exception ex)
           {
              //if meet this error, the server has been restored, we need to recover client database
              if (ex instanceof com.sybase.persistence.SynchronizeException)
             {
                  if (((com.sybase.persistence.SynchronizeException) ex).getErrorCode() ==  
                         com.sybase.persistence.SynchronizeException.SQLE_UPLOAD_FAILED_AT_SERVER)
                {
                     recoverClientDatabase();
                }
             }
           }
      }
      
      private void isRecoverFailed()
      {
          String dbFile = End2end_rdbDB.getConnectionProfile().getProperty(“databaseFile”);
          String recoverDbFile = dbFile + ".recover.ulj"; 
          if (new File(recoverDbFile).exists())
          {
             //todo 
             //implement to copy recoverDbFile content to recover dbFile
             return true;
          }
           return false;
      }
      
      private void recoverClientDatabase()
       {
          String dbFile = End2end_rdbDB. getConnectionProfile().getProperty(“databaseFile”);
          String recoverDbFile = dbFile + ".recover.ulj"; 
          //todo 
          //implement to copy dbFile content to recover recoverDbFile
      
          //retrieve all the subscriptions from client database
          GenericList<CustomerWithParamSubscription> _customerWithParamSubscriptions =               
          CustomerWithParam.getSubscriptions();
          GenericList<SISSubscription> _sisSubs = SISSubscription.findAll();
          GenericList<String> syncedPublication = new GenericList<String>();
      
          // check all the synchronization group, if is synchronized, add to new sync group to synchronize
          if (End2end_rdbDB.isSynchronized("synchronizationGroup"))
          {
              syncedPublication.add("synchronizationGroup ");
          }
          //retrieve  all local BO from client database
          GenericList< LocalMbo > localBoList = LocalBo.findAll();
          End2end_rdbDB.closeDBConnection();
          //subscribe with new database file
          End2end_rdbDB.deleteDatabase();
          End2end_rdbDB.subscribe();
      
          //merge old local BO data to new database
          for(LocalBo lc :  localBoList)
          {
              LocalBo localBo = new LocalBo ();
              localBo.copyAll(lc);
              localBo.create();
          }
      
          //add all the subscriptions from old database to new database
          for (CustomerWithParamSubscription sub :  _customerWithParamSubscriptions)
          {
              CustomerWithParam.addSubscription(sub);
          }
          for (SISSubscription sub :  _sisSubs)
          {
              ISynchronizationGroup sg = End2end_rdbDB.getSynchronizationGroup(sub.getSyncGroup());
              sg.setEnableSIS (sub.getEnable());
              sg.save();
          }
          //do sync
          String syncgroups = "";
          for(String pub :  syncedPublication)
          {
              syncgroups += pub + ",";
          }
          syncgroups = syncgroups.substring(0, syncgroups.length() -1 );
          End2end_rdbDB.synchronize(syncgroups);
      
          //finally delete the backup recover database file 
          new File(recoverDbFile).delete();
      }
    3. Resending local transaction
      This is a complete recovery method. Both methods above lose the local transaction. To prevent the lose of the local transaction when encountering the SQLE_UPLOAD_FAILED_AT_SERVER exception, the SAP Mobile Server Administrator must remove the client remote id information. The Administrator can locate the remote id from the server's mlsrv_err.log and call the ml_delete_remote_id procedure in the CDB to remove the remote id. The user can then continue to synchronize using the old database to upload all pending operations. But once uploaded, the user\application must recreate the database using either of the two methods described above, and must not reuse the old database anymore. The mlsrv_err.log logs remote id errors similar to this:
       I. 2013-04-14 14:13:39. <3> The sync sequence ID in the consolidated database:
              95bd47691098419cbf8539e8151bcf00; the remote previous sequence ID:
              95bd47691098419cbf8539e8151bcf97, and the current sequence ID:
              401be536e6e7417fb01b196276ec11c2E. 2013-04-14 14:13:39. <3> [-10400] Invalid sync sequence ID for remote ID
              'ed2ae448-a597-4f17-ad72-c6c61a6075a5'      
  3. Client application RBS beginSynchronize recovery
    beginSynchronize is an async pattern, requiring the user to override the com.sybase.persistence. DefaultApplicationCallbackHandler class onSynchronize method to check the SQLE_UPLOAD_FAILED_AT_SERVER error using the same three methods as described above to recover the client database. This sample code uses the second method and implements the AsyncCallbackHandler:
    if(isRecoverFailed())
    {
         recoverClientDatabase();
    }
    else
    {
        GenericList<String> syncList = new GenericList<String>();
        syncList.add(“default”);            
        synchronize(syncList);
    }
    
     private void isRecoverFailed()
     {
        String dbFile = End2end_rdbDB.getConnectionProfile().getProperty(“databaseFile”);
        String recoverDbFile = dbFile + ".recover.ulj"; 
        if (new File(recoverDbFile).exists())
        {
           //todo 
           //implement to copy recoverDbFile content to recover dbFile
           return true;
        }
        return false;
    }
    
    private void synchronize(GenericList<String> syncGroup)
    {
        AsyncCallbackHandler callback = new AsyncCallbackHandler();
        GenericList<ISynchronizationGroup> sgs = new 
        GenericList<ISynchronizationGroup>();
        for(String sg : syncGroup)
        {
            sgs.add(End2end_rdbDB .getSynchronizationGroup(sg));
        }
        callback.userContext = System.nanoTime() + "";
        End2end_rdbDB.registerCallbackHandler(callback);
        End2end_rdbDB.beginSynchronize(sgs, callback.userContext);
    
        int waitCount = 0;
        while (!callback.asyncDone())
        {
            if (waitCount++ > maxWaitTime)
            {
                throw new Exception("Asyn relay test failed because no response returned from server after waiting for 60 seconds.");
            }
            try
            {
                Thread.sleep(1000);
            }
            catch (Exception e)
            {
            }
        }
    
        try
       {
             Thread.sleep(4000);
       }
        catch (Exception e)
       {
       }
    
       if (callback.errorMessage != null)
        {
            throw new Exception(callback.errorMessage);
        }
        callback.userContext = null;
    }
    
     private void recoverClientDatabase()
     {
        String dbFile = End2end_rdbDB. getConnectionProfile().getProperty(“databaseFile”);
        String recoverDbFile = dbFile + ".recover.ulj"; 
        //todo 
        //implement to copy dbFile content to recover recoverDbFile
        //retrieve all the subscriptions from client database
        GenericList<CustomerWithParamSubscription> _customerWithParamSubscriptions =               
        CustomerWithParam.getSubscriptions();
        GenericList<SISSubscription> _sisSubs = SISSubscription.findAll();
        GenericList<String> syncedPublication = new GenericList<String>();
    
        // check all the synchronization group, if is synchronized, add to new sync group to synchronize
        if (End2end_rdbDB.isSynchronized("synchronizationGroup"))
        {
            syncedPublication.add("synchronizationGroup ");
        }
        //retrieve  all local BO from client database
        GenericList< LocalMbo > localBoList = LocalBo.findAll();
        End2end_rdbDB.closeDBConnection();
        //subscribe with new database file
        End2end_rdbDB.deleteDatabase();
        GenericList<String> syncList = new GenericList<String>();
        syncList.add(“default”);
        synchronize(syncList);
    
        //merge old local BO data to new database
        for(LocalBo lc : localBoList)
        {
            LocalBo localBo = new LocalBo ();
            localBo.copyAll(lc);
            localBo.create();
        }
    
        //add all the subscriptions from old database to new database
        for (CustomerWithParamSubscription sub : _customerWithParamSubscriptions)
        {
            CustomerWithParam.addSubscription(sub);
        }
        for (SISSubscription sub : _sisSubs)
        {
            ISynchronizationGroup sg = End2end_rdbDB.getSynchronizationGroup(sub.getSyncGroup());
            sg.setEnableSIS(sub.getEnable());
            sg.save();
        }
        //do sync
        synchronize(syncedPublication);
        //finally delete the backup recover database file 
        new File(recoverDbFile).delete();
    }
    
    class AsyncCallbackHandler  extends  DefaultCallbackHandler
        {
             private volatile boolean asyncCompleted = false;
             private volatile boolean asyncUploaded = false;
             public volatile String userContext = null;
             public volatile String errorMessage = null;
    
             public boolean asyncDone()
            {
                 return asyncCompleted;
            }
    
            public  SynchronizationAction onSynchronize(GenericList<ISynchronizationGroup> groups,  
                    SynchronizationContext context)
           {
                Exception ex = context.getException();
                 if (ex instanceof com.sybase.persistence.SynchronizeException)
               {
                     if (((com.sybase.persistence.SynchronizeException) ex).getErrorCode() ==  
                            com.sybase.persistence.SynchronizeException.SQLE_UPLOAD_FAILED_AT_SERVER)
                   {
                        recoverClientDatabase();
                   }
               }
    
               if (context.getStatus() == SynchronizationStatus.ASYNC_REPLAY_UPLOADED)
               {
                  if (!End2end_rdbDB.isReplayQueueEmpty())
                  {
                      throw new Exception("need sync  is not correct!");
                  }
                  asyncUploaded = true;
              }
              if (context.getStatus() == SynchronizationStatus.ASYNC_REPLAY_COMPLETED)
              {
                  if (userContext != null && !userContext.equals(context.getUserContext())) //Not for this round
                  {
                      return SynchronizationAction.CANCEL;
                  }
    
                userContext = null;
                End2end_rdbDB.synchronize("so");
                if (!asyncUploaded)
                {
                    errorMessage = "ASYNC_REPLAY_COMPLETED is received without ASYNC_REPLAY_UPLOADED";
                }
                asyncCompleted = true;
                return SynchronizationAction.CANCEL;
            }
            return SynchronizationAction.CONTINUE;
         }
     }
  4. MBS for DOEC client Application recovery
    MBS for DOEC clients run on C# and Android platforms. The registration recovery is the same as RBS registration recover. But the Subscribe recovery is different. There are two scenarios:
    • SAP Mobile Server restored to a point-in-time where the client unsubscribed – after SAP Mobile Server is restored, the client status is unsubscribed on the server-side, But the client status may be subscribed or unsubscribed. The client application should check the client subscription status; if subscribed, call unsubscribe(this leads to unsubscribe failure message because the server does not have client subscribe information), then call subscribe. If unsubscribed, call Subscribe. Sample code:
       If(MyDatabase.isSubscribed())
      MyDatabase.unsubscribe(); 
      _rh.waitForMessage("UnsubscribeFailure", "UnsubscribeSuccess"); 
      MyDatabase.subscribe(); 
      _rh.waitForMessage("SubscribeSuccess");               
    • SAP Mobile Server is restored to a point-in-time where the client subscribed – after SAP Mobile Server is restored, the client status is subscribed on the server-side. In this scenario, the client can check if it is subscribed and, if not, call Subscribe to recover. After SAP Mobile Server is restored, the SAP Control Center Administrator can also delete the subscription from server-side. This time, SAP Mobile Server sends an unsubscribe message to the client application. When the client application is running, it receives the unsubscribe message, at which time the client generated database calls DB.cleanAllData() to clean the client data and update the status to unsubscribed. The client application needs to call subscribe:
      Thread.sleep(10 * 1000); 
      //sleep some time to receive server unsubscribe message when application start up
      If(!MyDatabase.isSubscribed())
      MyDatabase.subscribe();