Netbackup Appliance 3.1.1 (8.1.1.) crashing
After upgrading Netbackup Appliance 5230 from 3.1 to 3.1.1 (Netbackup 8.1.1) we have had frequent unexpected reboots. The OS reboots and there is nothing in the Netbackup logs explaining what went wrong. Also there are no core dumps created. The only bit of detail is this strange behaviour in the /var/log/messages, below, note the break sequence ^@^@ - looks like the OS runs out of kernel(?) memory even though the available memory is almost 50% - by checking sar reports eg. sar -r -f /var/log/sa/sa{dayofthemonth} After weeks of struggling with support we eventually found an engineer that dug up a known bug! for instability in Netbackup 8.1.1. The EEB is ET3942191 but please note that 8.1.1 already comes with this EEB installed but Veritas have silently being updating it and not telling anyone - nowhere to be found in late breaking news - shocker I know...:smileyindifferent: If I remember correctly 8.1.1 comes with version 1 or 2 of this EEB. My engineer gave me version 9 of it. Once it was installed the rebooting went away for a month. A new case was logged when the issue re-occured, I then after being told by support there is nothing they can do for me, I asked them about ET3942191 because maybe it was updated again, and low and behold it was! I was given version 14 to install and its been stable since - holding thumbs. Patch installed was NBAPP_EEB_ET3942191-3.1.1.0-14.x86_64-180814140122.rpm. So if you have weird crashing on your Netbackup Appliance running version 3.1.1 ask for ET3942191 it may change your life and award you extra sleep at night - might also cure baldness. I am hoping this tyrade of mine will help someone that is experiencing the same thing. May 28 16:50:09 netbackup-appl-01 systemd: Started Serial Getty on ttyS0. May 28 16:50:09 netbackup-appl-01 systemd: Starting Serial Getty on ttyS0... May 28 16:51:09 netbackup-appl-01 systemd: serial-getty@ttyS0.service holdoff time over, scheduling restart. May 28 16:51:09 netbackup-appl-01 systemd: Started Serial Getty on ttyS0. May 28 16:51:09 netbackup-appl-01 systemd: Starting Serial Getty on ttyS0... ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@May 28 16:54:10 netbackup-appl-01 kernel: Initializing cgroup subsys cpuset May 28 16:54:10 netbackup-appl-01 kernel: Initializing cgroup subsys cpu May 28 16:54:10 netbackup-appl-01 kernel: Initializing cgroup subsys cpuacct May 28 16:54:10 netbackup-appl-01 kernel: Linux version 3.10.0-693.17.1.el7.x86_64 (mockbuild@x86-041.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Sun Jan 14 10:36:03 EST 2018 May 28 16:54:10 netbackup-appl-01 kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-693.17.1.el7.x86_64 root=/dev/mapper/system-root ro nomodeset rdblacklist=qla2xxx rd.lvm.lv=system/root rd.lvm.lv=system/swap nodmraid LANG=en_US.UTF-8 crashkernel=256M loglevel=1 pcie_aspm=off modprobe.blacklist=qla2xxx,ahci,isci audit=1 May 28 16:54:10 netbackup-appl-01 kernel: e820: BIOS-provided physical RAM map: May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000008bbff] usable May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x000000000008bc00-0x000000000009ffff] reserved May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x0000000000100000-0x00000000bad06fff] usable May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x00000000bad07000-0x00000000baf83fff] reserved May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x00000000baf84000-0x00000000bafb5fff] usable May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x00000000bafb6000-0x00000000bafcafff] reserved May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x00000000bafcb000-0x00000000bb3d3fff] usable May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x00000000bb3d4000-0x00000000bdd2efff] reserved May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x00000000bdd2f000-0x00000000bddccfff] ACPI NVS May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x00000000bddcd000-0x00000000bdea0fff] ACPI data2.4KViews5likes1CommentNbServerMigrator Tool
Hello NBU world! I've just completed two master server migrations using the NbServerMigrator tool (HP-UX > RHEL), and I was curious to see if anyone else had experience with this tool? Veritas has a pretty comprehensive document to using it, but I still encountered issues that were not documented. Aside fromthis guide,there is a lack of documentation related to its usage. As NBU 8.1.1 nears its EOL, I think there will be an influx of administrators using this tool to migrate off of HP-UX and AIX servers. I've documented the issues I experienced and how they were resolved, but I wanted to receive input from any other admins who have used this tool before compiling a TechNote or forum post. I'll be posting either a TechNote or forum post with more detailed info, but I've pasted a general overview of the issues I encountered below. Issues I encountered: 1. In both migrations, the pre-check did not flag the root/sys user/group combo being required. Once we began the migration, the transfer failed and told us to fix that. 2. The data transfer takes a LONG time. There is some area for improvement here for how I performed the migration (used default compression and other settings, which could have minimized the transfer time if tuned better). One domain had a catalog ~250GB in size, and the other was ~1 TB in size. 3. The documentation says that the -clean_up switch removes ALL temporary data, including the image data that had already been transferred. I had to perform this twice due to interrupts in the transfer, and neither time caused the tempdb directory (which contains the image information) to be removed. 4. Both migrations required quite a bit of manual intervention to get the certificates straightened out on the target servers. Once you perform the final -overwrite step, the source sends its certificates to the new master, which already has certificates configured as part of the initial NBU install. This part (after a 12+ hour migration), is quite painful, as it is one of the last steps before the migration concludes. 5. During the second migration, the terminal session between the target and source was terminated due to an ISP outage by the person helping me with the migration (we were sharing screens on his computer). This caused quite a lot of trouble with the migration (I'll detail it in my in-depth write up). There are a few more things I observed during the process, but I was curious if anyone else had observations or experience with this tool.1.4KViews2likes3CommentsNetbackup admin console ONLY Backup Archive and Restore
Admininstration Console only show Backup Archive and Restore I've solved a thing that other can have use for. Regarding the auth.conf.win.template (which should be use for auth.conf). Problem: My colleauge installed a ner Netbackup Master Server with default settings (no fancy pancy). He had full rights and all looks really nice, until I was suppose to be supriced of the ince gui with all access. There was only BAR (Backup Archive and Restore). Facts: We have no auth.conf and log / admin confirmed not root access.. hmm But we did have auth.conf.win.template Solved: In the auth.conf.win.template i found on our windows that is was singel lined (no CR carrige return) on: domain\userADMIN=ALL JBP=ALL* ADMIN=JBP JBP=ENDUSER+BU+ARC When open notepad with administrator rights and see that the auth.conf.win.template had a CR and saved the file and restarte the mighty GUI: domain\user ADMIN=ALL JBP=ALL * ADMIN=JBP JBP=ENDUSER+BU+ARC Tada (sound effect). it worked!939Views2likes0CommentsSolaris 11 and SL8500 - master media servers won't boot!
Just a FYI to anyone with SL8500 and Solaris 11 media servers I have Solaris 11 master/media servers and write to a SL8500 tape drive. Generally works like a champ! Last weekend, I did a reconfigure reboot on my media servers to update the tape drives - and it would not connect to the network! saw errors like this: svc.startd[13]: svc:/network/datalink-management:default: Method or service exit timed out. Killing contract 12. svc.startd[13]: network/datalink-management:default timed out: transitioned to maintenance (see 'svcs -xv' for details) Clearly a network issue. Resolution - disconnect FC to SL8500 robot! We ended up unzoning it instead, since it is not actually the FC that is causing the issue. The SL8500 looks fine, the OS is fine, just cannot connect them together. Foundhttps://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=297830852382714&id=1970275.1&_afrWindowMode=0&_adf.ctrl-state=qmsn6ikpa_716 Eventually powered off the SL8500 and powered it back on again, problem solved. Two full days of Oracle Sev 1, with no appreciable assistance, ended up solving it ourselves.549Views2likes1CommentSLP - Pro Tip - Use Storage Unit Groups where possible!
Just migrated in some new media servers. I had to update several hundred SLP policies Created some storage unit groups, like Tape Group 1, Tape Group 2 and pointed the SLP at the storage unit groups. Next time I update my tape media servers, I can just update the Tape Group with the new media server, instead of my SLP. Way less work!767Views2likes1CommentNetBackup GRT Support for Exchange 2016 CU5
We recently issued EEB 3918337 for NetBackup 8.0 to support Exchange 2016 CU5. Database backups and restores are fine without the EEB, as are GRT backups. You need the EEB to browse and restore mailbox content from your backups. The EEB is an adaptation to changes in the Exchange database. Future NetBackup releases will include the adaptation. Earlier Exchange versions such as 2016 CU4 continue to be supported with or without the EEB.Solved3.3KViews2likes8CommentsCertificate on master server expired
i have a problem with a NB master server 8.0 which was not used for more than one year. Now i cannot connect since the security certificate is expired. It is the master server itself. I tried to renew on the server with "nbcertcmd -renewCertificate" but without success. Then i tried to revoke the certificate and reissue with a generated token but to do so i have to login to WEB Mangement service with "bpnbat -login -logintype WEB". But this is also failing. (Login to Authentication Broker only is successful but does not help) Any help is appreciated thanks KaiSolved9.1KViews1like11CommentsBMR via Hyper-V NVGRE works
FYI - just in case anyone else wants or needs to know if this works, and proven that it does... NetBackup Master Server v8.1.1 on Windows Server 2016 NetBackup Media Server v3.1.1 on Appliance 5340 - MSDP storage hosting hypervisor: Windows Hyper-V Server 2016 Cluster client: NetBackup Client v8.1.1 on Windows Server 2016 client networking: client residing within an NVGRE bubble backup: full client backup via Windows based agent within guest VM, with BMR and TIR enabled restore / recovery: whole VM via BMR, across NVGRE - this worked just like a normal non-NAT'd VM, and certification with a re-issue token worked, with the end result of a guest VM fully rejoined to its original AD domain within the NVGRE bubble.1.3KViews1like1Comment