For PSC’s most recent system install, the team took a collaborative & strategic approach, enabled – not hindered – by COVID-19.
Senior Network Engineer Rich Angeletti still refers to himself as a “new” staff member after 6 years at PSC. After all, he’s the newest on a team of engineers, many with 20-25 years of experience at the center, and the stories to prove it. But Rich’s status as a rookie may just be starting to change after serving as an integral part of the Bridges-2 installation team, PSC’s most recent large scale system installation to date. In his time at PSC, Rich has been part of the team for both the Bridges and Bridges-2 installations, and was able to observe the great strides our teams took to make the Bridges-2 process a success.
We were able to apply a lot of lessons learned from the Bridges installation over 5 years ago for the Bridges-2 installation. Things also tend to change with each new system install and we are constantly adjusting for the unique considerations of each new computing platform. “For this installation, we had to make room for the system, and we had to move a lot of things around in the data center and make some major electrical upgrades, so the scope of work ended up involving every group,” Rich said.
The installation of Bridges-2 came at a complicated time, with COVID-19 keeping nearly all PSC team members at home. Certain staff were granted permission to work in the data center with the physical machinery, but beyond this, our teams coordinated the installation virtually. “Due to the nature of remote work, people needed to work together more and communicate more often and more effectively.” According to Rich, the nature of the Bridges-2 installation required a massive team effort over Zoom and phone calls. He told us, “people have been doing a great job of working across groups and working collaboratively.” Just like so many other businesses in the past year, PSC has not just adapted to the “new normal,” but has sought out opportunities to help our team grow despite the challenges it has presented.
It’s clear from her detailed descriptions and precise explanations that Anjana Kar is an expert in her field. She is a Senior Storage Systems Specialist, and worked on the installation team to provision the nodes on Bridges-2, among other – highly complex – contributions. It came as no surprise, then, when she shared that she joined PSC way back in 1988. Even with her breadth of experience, Anjana has typically worked on storage systems and maintenance work, rather than installation projects from the beginning. But her experience working on predecessor Bridges provided her with many insights to share about the progress made between the two massive projects.
Similarly to Rich, Anjana was impressed by the teamwork on display throughout the Bridges-2 installation process – teamwork which is still ongoing as Bridges-2 gets online and into production. She has gotten to work with teams, that even in her decades with PSC, she’s had little experience working closely with.
The pandemic provided challenges that the team leveraged into opportunities: remote work gave way to more collaboration across distinct groups, and shipping delays created a longer timeline, which in turn led to additional testing and an even better machine being ready for our users in early 2021.
According to Anjana, the Bridges-2 hardware was due to arrive in July 2020, but ended up arriving in November. Despite a less than ideal circumstance for our hardware installation team, not to mention our users eager to get working on the exciting new Bridges-2 platform, our team took advantage of this extra time. Anjana, working with colleagues Brian Johanson and Ryan Sablosky, ran extra tests and configured the operating system for Bridges-2. The original Bridges machine provided a great roadmap for the team – both of what to do, and what not to do, guiding the team along the way as they fine-tuned the software. After the machine was tested onsite by HPE and handed over to PSC on December 1, the software installation team was beyond ready, and in about two weeks staff were able to run tests and start their work ahead of letting in users.
Throughout our conversation, Anjana emphasized how “smooth” the entire process was. By utilizing the extra time due to shipping delays to run more tests, coordinate with the requisite teams, and get the data center ready for its new machine, the team not only avoided any major hiccups, but got a machine up and running that was going to provide the best possible user experience.
The list of people involved in Bridges-2 comprises almost every member of staff at PSC. The Slack channel for Bridges-2 includes 41 people – out of PSC’s 60 employees – and that probably doesn’t include everyone who has been involved in the project to date. As Project Manager Amanda Slimick puts it, “everyone who was involved with Bridges-2 was willing to go above and beyond for this project. It was a big team effort with everyone from [our director] Shawn down, getting involved and lending a hand at all stages.”
With this many staffers involved, it goes without saying that a new system install involves more than just the physical hardware installation or the software configuration. The team members involved on Bridges-2 have worked on a variety of installation activities, including: upgrading networking switches for free flow of network traffic; creating user documentation; updating training materials for users; installing and configuring system software; deploying system monitoring tools; executing tests to establish system benchmarks to ensure functionality; and implementing safety protocols in our machine room for those team members who needed to work on-site during COVID-19.
Everyone involved with Bridges-2, which is nearly all of PSC, will attribute the success of the project to their fellow teammates when asked about it. This is not unique to Bridges-2, but in fact an element of PSC culture of which we are extremely proud. However, what was unique to the Bridges-2 installation was how much our teamwork and collaboration played a role in the success of the project, due to the challenges presented by COVID-19. By turning challenges into opportunities, the staff were able to ensure a smooth process from start to finish, and we have a machine in production that will enable groundbreaking research for years to come.