Forum Discussion

steves_2's avatar
steves_2
Level 4
4 years ago

Backup exec 21.2 Upgrade from 21.1 Speed Issue

Hi All,

Has anyone noticed any speed issues with BE 21.2 (21.0.1200.1899)

For example a backup of a VM that takes 40 seconds now takes over 4 mins, used to take 1 min with 21.1.

Look at the start and finish times compared to the VM backup times.

 

21.2

Backup started on 12/05/2021 at 17:01:45.
Backup Set Detail Information 
Media Label: IMG000170
GRT backup set folder: E:\BEData\IMG000170
The standard processing method will be used for incremental and differential backups.
The data in Pagefile.sys will be skipped during backup because the data is transient and does not need to be restored.
Backup Exec has discovered and protected 'C:' on virtual machine 'XXXXXX-DC01.XXXXXX.local'.
Backup Exec has discovered and protected Microsoft Active Directory data on virtual machine 'XXXXXXl-DC01.XXXXXX.local'.
Backup completed on 12/05/2021 at 17:05:50.
Backup Set Summary 
Backed up 3 files in 8 directories.
Processed 16,599,073,988 bytes in  40 seconds.
Throughput rate: 23745 MB/min

 

 

 

21.1

Backup started on 08/05/2021 at 17:01:07.
Backup Set Detail Information 
Media Label: IMG000114
GRT backup set folder: E:\BEData\IMG000114
The standard processing method will be used for incremental and differential backups.
The data in Pagefile.sys will be skipped during backup because the data is transient and does not need to be restored.
Backup Exec has discovered and protected 'C:' on virtual machine 'XXXXXX-DC01.XXXXXX.local'.
Backup Exec has discovered and protected Microsoft Active Directory data on virtual machine 'XXXXXX-DC01.XXXXXX.local'.
Backup completed on 08/05/2021 at 17:02:08.
Backup Set Summary 
Backed up 3 files in 8 directories.
Processed 16,210,747,454 bytes in  37 seconds.
Throughput rate: 25070 MB/min

 

 

I have seem something strange in the log files between the 2 backups.

In 21.1 i'm seeing this :

[7060] 2021-05-01T17:02:04.290 [loops] - BackupBSDProcessor::ExtendedAddFile: m_bVirtSet 1 m_virtDataId 60 m_virtBedsId 60 m_virtBedsVer 0
[7060] 2021-05-01T17:02:04.295 [server] - MSG_END_VIRTUAL_BACKUP_SET: Restore Work Flow (RWF)

in 21.2 im seeing this :

BENGINE: [05/12/21 17:02:41] [5800] [loops] - BackupBSDProcessor::ExtendedAddFile: m_bVirtSet 1 m_virtDataId 14 m_virtBedsId 73 m_virtBedsVer 0
BENGINE: [05/12/21 17:02:50] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:02:55] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:00] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:05] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:10] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:15] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:20] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:25] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:30] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:35] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:40] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:45] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:50] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:55] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:00] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:05] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:10] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:15] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:20] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:25] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:30] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:35] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:40] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:42] [5800] [server] - MSG_END_VIRTUAL_BACKUP_SET: Restore Work Flow (RWF) is empty

 

I'm getting a lot of these ndmp readit errors, what ever they are. I don't have ndmp enabled in the options.

Can anyone shed any light on what is going off here as i'm guessing this is causing all the VM backups to slow down making the backups times about 40% longer.

cheers

 

 

21
Windows Server 2012 R2
  • steves_2's avatar
    steves_2
    3 years ago

    HI, Just an update on this issue. https://www.veritas.com/content/support/en_US/article.100051020

    Last week Veritas release Backup Exec 21.4.

    This fixes the issue i was having with slow back times after upgrading to 21.2.

    My Backups are now completing roughly 30% faster with 21.4 bringing it back inline to what i was seeing with 21.1.

    Issue(s) resolved

    • Instant GRT taking time post upgrade to BE 21.2

    Cheers

    Steve

  • CraigV's avatar
    CraigV
    Moderator
    4 years ago

    Hi,

    I've seen this on previous versions where a version that was updated too ended up being slower. Make sure there are no new driver requirements for your backup target, and also check LiveUpdate to see if there are any updates available for the version you are now on.

    Thanks!

    • steves_2's avatar
      steves_2
      Level 4
      4 years ago

      Hi Craig, thanks for the reply.

      Nothing extra required that i can see in the guide or read me. I even found and updated to BE 21.2 FP2 (21.0.1200.1930) to see that that made a difference, but it hasn't.

      It just looks like a ndmp issue with the new version and it is timing out, putting huge delays in the backups as it retries every 5 seconds.

      It must be something to to with the 21.2 backup engine as some VM's i haven't updated the remote agent and they are showing the same slower backup running on  21.1 agent.

      Cheers

       

      Cheers

      • NetworkCompany's avatar
        NetworkCompany
        Level 4
        4 years ago

        I noticed this also since R2.  Performing an agent based backup runs for us at ~25,000MB/min while Hyper-V based backups of the same VM barely achieve 3,000MB/min.  Haven't been able to point at exactly what the difference might be yet here.

  • NageshPatil's avatar
    NageshPatil
    Moderator
    4 years ago

    We have not seen any concerns about slowness in VM backup after upgrade to Backup Exec 21.2. We suggest we check below things to possibly isolate the cause : 1. Does Backup Exec Remote Agent version of all VMs matches to the media server version i.e. 21.2 ? If not, I suggest we upgrade all remote agents and observe the speed. 2. In case you are observing the slowness for specific VM, I suggest you to disable all GRT options and give a try once. This would help to isolate any VM or workload specific issue. Thank you. Regards, Nagesh

    • steves_2's avatar
      steves_2
      Level 4
      4 years ago

      Hi,

      I've done some more testing today.

      I exported a small VM (15GB) from our system to a new test server and installed 21.1

      This server has backup exec and hvper-v installed so there is no network traffic as everything is local.

      Ran the backup with debug logs enabled and catalog generation took 42 seconds on 21.1

      [14052] 2021-06-25T10:02:12.989 [loops] - PDI backup: Starting catalog generation on server [TestHV]

      [14052] 2021-06-25T10:02:54.208 [loops] - PDI backup: Catalog generation done

      I upgraded straight to 21.2r2 and ran the backup with debug logs enabled and same catalog generation took 3 mins and 48 seconds.

      [8408] 2021-06-25T11:02:11.799 [loops] - PDI backup: Starting catalog generation on server [TestHV]

      [8408] 2021-06-25T11:05:59.772 [loops] - PDI backup: Catalog generation done

      It's all seems to be pointing to the Instant GRT cataloging for VM's.

      This particular VM was a domain controller, but i'm seeing the same slowness with SQL databases as well.

      Disabling GRT on 21.2r2 does make it run faster, but still not as fast as 21.1 with instant GRT.

      So i'm seeing exactly the same issue on different server with backup exec with backup exec 21.1 v 21.2.