vSphere 6.0 -Deprecated VMFS volumes

After adding some SAN storage to my UCS Blades, they displayed an error in vSPhere that said “Deprecated VMFS volume(s) found on the host.  Please consider upgrading volume(s) to the latest version.

My volumes were already the latest and greatest at VMFS5 so I turned to trusty google.  Apparently this is a known issue affecting vCenter Server 6.0.  It said to restart the management agents on each host to clear up the issue, sweet.  I logged onto each host using the DCUI and proceeded to restart the management agents…the error never disappeared.  Grr.  Digging a little deeper I found that some people who fixed it were doing it through the command line.  Both methods should do the same thing but I gave it a shot.  Sure enough, connecting to each host via putty (ssh) and running the “services.sh restart” command did the trick.  It may throw some alerts but you shouldn’t lose connectivity to anything and the alert will promptly disappear.

For more information on this error see vmWare’s support:




Cisco UCS – Unable to boot to SAN

During the configuration of my UCS B-Series I ran into some issues with booting to SAN that required me to open a TAC case after exhausting the Internet.  The answer was so counter-intuitive that I wanted to post it here in case anyone else can use it.  Here are the steps I took and where I hit a roadblock.

First I created host profiles in the UCS for each blade and applied WWNs from the SAN pool.  Each blade has a side A and side B WWN which I zoned to different node pairs on our 3par to distribute the load.  Next I cabled the UCS to our fabric switches and set up the boot policy on each blade to boot to one of the WWNs from each side of the fabric, one for the primary and one secondary.  I created boot volumes on the 3par for each blade and powered up the blades.  I set them to boot to a locally mapped ISO and when the prompt came to scan for storage to use for the install, they all found their proper disks immediately.  After rebooting the blades could not find their boot disks.


The catch is that there are 4 storage WWNs mapped on each side, so you can’t just pick any WWN.  To find out what one you need to use, you will have to get onto the command line of the controller on each side of the UCS and run these commands.

You’ll need to ssh using putty (or any ssh client) to each controller of the UCS and log in as an admin.

Type: Connect Adapter 1/1/1
This command is used to identify each blade, just change the middle number to correspond to each blade you need to pull up.  Blade 2 would be 1/2/1 and so on.

Type: Connect

Type: attach-fls

Type: vnic
Here you will see two listings, and you choose the one you want to view in the next command.  Choose the top if you’re on side A, choose the bottom if you’re on side B.


Type: lunlist xx
Where xx is the number of the vnic you want to list, choose the top one if you’re on side A, the bottom if you’re on side B.

What you’ll see listed here is a list of all of the WWN paths currently zoned.  The top number is the one you need to enter on the first graphic above for the boot order SAN targets.  These are the only ones that will work!

Hope this helps!  If you have any questions please feel free to drop me a comment.

Veracity Coldstore Review

My company has utilized Veracity Coldstores for about 4 years at a large public event space exclusively for archiving security camera footage for around 300 cameras.  These storage devices are specialized, less expensive than standard enterprise storage and extremely durable.  Part of their design allows them to shut down hard drives that are not in use, greatly extending their lifetimes.  We actually haven’t had a single drive fail in my tenure.

Our first products in 2012 were the dark blue Arcus 4U models.  We loaded them up with 15 Purple WD 4TB drives that are made specifically for surveillance per Veracity’s recommendation.  Using a centralized security camera management software (Genetec Security Desk) we use a local Windows drive for dumping the video footage temporarily.  Veracity’s Video Mover app continuously scans the drive and picks up files to move over to its own storage.  At about 50 cameras per Windows server, with 6 servers sharing 3 Coldstores on 4gb fibre it keeps up with the load just fine.  Our cameras are on continuous record so there’s never any downtime.

We’ve also used them in conjunction with Avigilon camera software, but only as an extended archive.  Avigilon uses local storage as a best practice, and when we tried backing up footage from an Avigilon VM to any other drive, we lose most of the advanced browsing/zooming functionality so our security department asked for it to be kept locally.  We ended up doing backups of footage older than 30 days to the Coldstores with Video Mover and it works great in that capacity.

We are currently setting up a new facility of a similar size with their new Coldstore 3U devices and 6TB purple drives.  They look sharp with a brushed aluminum finish and smaller size.  We received service above and beyond the line of duty from Veracity as we needed to get the cameras going before the datacenter was completed.  Their installers were required to undergo security training and work in dusty and dangerous conditions but still managed to get our Coldstores racked and stacked ahead of schedule.  Great company to work with and their product works so well you never have to think about it.




HPE Storage Symposium 2016

This is my review of the 3 day HPE Storage Symposium which is a small yearly event in Colorado Springs, CO.  Sharing a city with USAF and a few lingering tech companies, this is the former site of an HPE corporate office but only the conference remains. We had the luxury of staying in 5 star accommodations at The Broadmoor.

Day 1
Travel and accommodation check-ins were as smooth as could be with some weather added in at DFW.  The Broadmoor shuttle was a place to meet some new friends who were fellow guests from around the country.  They had a cocktail and dinner event on the first night and it was a short walk from the main Hotel building around a gorgeous lake to the reception.  I met some folks from the Detroit area who were in medical tech and we joked about fun with HIPAA and a lot of guests were asking about my well-known company.


That evening I came back to a gift in my room, a Broadmoor black dry-fit pullover jacket with an HPE logo on the bottom…it looks sharp and I still wear it often.

Day 2
The actual day of presentations started bright and early with breakfast at 7am in the main hotel building, although I started off with a 5am workout in the surprisingly happening spa/gym.  I was well pampered with a tv on my treadmill, headphones, and fruit infused water.  For breakfast there were plenty of healthy options along with the standard scrambled eggs, french toast variety.

One of the standout speakers was Jennifer Welch, an impressive technical leader with HPE who mapped out future innovations and ideas, and even touched on some shortcomings of the past that have been reconciled.  She was engaging and fun, and I admired her poise in a room full of critical, technical men who seemed to accept her as one of their own by the end of her talk.

Another excellent speaker was AJ Castameto from Brocade.  Since we’ve made the decision to switch from Cisco SAN switches to Brocades at our new datacenter I was very interested in what he had to say.  He knew better than to sound sales-ish and instead focused on subjects such as lessons from the cloud:
Do you trust infrastructure that you can’t control?  A lot of companies I know of have dipped their toe in the water only to quickly pull it back.  A few have completely jumped in and drowned.  I have never been a fan of putting our data out into a foreign data center simply for convenience or cost savings.  But then again I have always been fine doing things the hard way if I think it will save me effort in the long run.  Short term fixes are just that, and in my current role I am confident that I’ll be the one fixing it in a couple years when it breaks anyway. AJ was engaging and right on with his advice on data center stability and the importance of analytics from your devices, and even if it wasn’t all news to me it was great to be in good company.

After everyone’s initial talks we went to lunch and came back for “deep dive” sessions with the same speakers.  This allowed them to get more technical and granular with their areas of expertise and it held my attention easily.

That evening a gathering was planned with a 5 course sit-down dinner that was top notch.  Unfortunately I sat next to a fellow who was older and from one question about his cast proceeded to tell me his full medical history and rehab plans for his rotator cuff.  I left as soon as I could and went back to my room to relax.

I came back to a nice surprise in my room…a gift bag with an Avnet bath robe and other goodies.  I couldn’t ask for nicer hosts.

Day 3
Our hosts gave us the option to play a round of golf on the amazing course surrounding our hotel, or to visit the spa for an hour-long massage.  My schedule was a little tight with my shuttle leaving at 10am but everything worked out perfectly and my massage was a great way to wrap up my trip!  The staff at The Broadmoor could not have been nicer and my room couldn’t have been more comfortable with a view of the lake pictured above.  I’ll definitely attend next year if I’m invited by our vendor again.