First Off, I want to thank Gary for giving me an opportunity to be a guest writer on his blog, it’s an honor. My name is Dan Chilton and I have worked in technology for the past 20 years. As an introduction, today I just want to tell the story of why I became a performance engineer. . .
My first lucky break job working in IT came in my late 20s. It was 2003 and the IT job market was still recovering from the dot com implosion. After 2 years of blundering my way through an MCSE on Windows 2000 and more practice tests then I care to admit I passed the 5 exams. My first real tech gig was Tech Support Engineer level 1 at Network Appliance. It was a lucky break at that time they had no Level 1 position. They created the role because I kept asking if they would give me a shot until they finally did. This was back in the days when Network Appliance logo was a giant nut flying toward a bolt. It was several years before they would become NetApp, rebranding themselves with a post-modern smurf blue Stonehenge logo.
Network Attached Storage was a good place to be at that time as data was already growing exponentially each year. I was given the opportunity to learn Storage on the job and I dove into my trainings and cases in earnest. They say necessity is the mother of invention. There is nothing like being the Tech Support Engineer on the phone who is supposed to have all the answers. Speaking to customers will force you to learn storage and learn it quickly. Staying one step ahead on a support call was a terrifying experience. Then again, so was bartending at a Bourbon Street club but that’s a story I will leave for another occasion. I prayed a lot, learned a lot and eventually worked my way up to Tech Support Engineer Level 3. The base of knowledge that was required was wide. You had to work cases including CIFS, NFS, SAN, hardware, down filers, metroclusters, snapmanagers, performance, backup, and multi-protocol permissions. It was this final case type multi-protocol permissions that I think drove me to become a performance specialist.
Windows servers connect to shares over SMB (back then it was CIFS). Access to data is controlled by user and group membership. It’s a complicated (Why is Windows always so complicated?) set of file, folder, and share permissions applied when users want to touch their data. This is fundamental Microsoft Windows Admin stuff and it was necessary for every filesystem supporting Windows clients whether NetApp ONTAP or EMC. On the other side was Linux (back then it was Unix) connecting to NFS shares. The permissions style was and still is user, group, and everyone else and read, write, and execute. One key value of NetApp was that it created a NAS that was multiprotocol of CIFS and NFS allowing both user types and permissions styles to access. But here is where it gets really complicated. An admin could set on a volume either Windows style permissions, Unix style permissions or mixed mode. With Windows style perms, if a Windows user needed to access Unix data, they could user map to a unix account to access unix data. User mapping was a pain in the butt, but it lent an orderly logic to the permissions style on the files. On unix style volumes, unix users could access unix data, or map to Windows user for Windows data. Sounds easy right? Given that this was painful to set up and manage, there was one more style called mixed mode. Mixed mode sounded like the ideal solution, with permissions for Windows and permissions for Unix data on the same set with no need for User mapping. Instead it wound up being ridiculously complicated, kind of like any trip to the department of motor vehicles. When mixed mode was used, the permissions on the file or folder changed based upon the last user to touch the file. Unix permissions would be applied after a touch by a Unix user or Windows user permissions after a Windows user touch. This lead to problems where users could access their data one day and then randomly not be able to access it the next. And it led to lots of confused and sometimes angry customer phone calls.
After I worked the 10th mixed mode permissions case, I was done. I was becoming an angry person, I didn’t like people anymore, I thought of going back to bartending. I knew I needed to specialize in something just to get out of the permissions business and I really liked the speed and complexity of performance, so that’s what I went after. I worked for 6 years at NetApp in workload engineering performance, followed by another 7 in the Nutanix core performance team. I still get up every day grateful to work on performance problems and not permissions problems.
I don’t pretend to have all the answers, but I am happy to spark discussion and learn things. As a guest blogger I have a couple of things I’d like to write about.
- Working in Nutanix Engineering
- Nutanix Files, my product area
- Performance benchmarks and workloads like specstorage 2020
Stay tuned!