Credit: Dreamstime
Two years ago, the enterprise team at the ANZ Group found themselves juggling an IT ecosystem which at times its enterprise team could best describe as a car wreck.
Working with 7,000 systems and 5,500 servers, the team faced the mammoth task of rigorously checking and enforcing the bank’s compliance across all of these at a cost of thousands of hours in manpower.
Seeking a way to speed up and simplify the process, ANZ turned to IT automation software vendor, Puppet.
“We started with 7,000 systems that were almost all different from each other, in lots of different operating systems,” explained Boyd Adamson, ANZ Bank senior consultant, enterprise platforms. “So we had Solaris, Linux - several different versions.
“There are about 22 different regulatory bodies in several different countries and they all have different sets of rules. All of those have to get configured down to a certain point into our operating system. Some of that includes what packages need to be installed and various security settings in various areas.”
“We had an environment that was everything from a lovely car, to a car wreck,” quipped Nathan Kroenert, a fellow senior consultant for the bank’s enterprise platforms.
Speaking at the Puppetize conference in Sydney earlier this week, the pair recalled how before hiring the US-founded vendor, banking compliance was largely managed manually by various teams.
“The way things worked before was that there was this huge script that was pointed at a server, would look at the compliance rules and then report it was wrong,” said Adamson.
“A human would then take this report and turn it into a number of service requests. Those requests would then be taken by another bunch of humans who would do their best to make the systems compliant.”
“And yes we were compliant, but it was not straightforward, simple, effective and scalable,” added Kroenert. “I didn’t want to sit down for six months with an auditor to explain why we went for this system to this statement of its compliance. If I’m saying it gets passed from this team to this team, that is not good for our organisation.
"The idea was to take all these servers, which were reporting only rules, give them a set of Puppet code that not only did the reporting, but not the implementation, and them to report and enforce them automatically.”
With Puppet on board, the enterprise team began the process of working manually through all the 7,000 systems, which, due to the bank’s heavy use of security-protected sub-networks - known as demilitarised zones - took some considerable time.
Once presented with the system, they then had to stand them up against the compliance rules and see if any changes or variations were needed. Only then could they begin rewriting the codes to turn them into “useful Puppet code” that would automate compliance enforcement of the server.
Starting with the easy codes, before tackling the more complex, the bank carried out the process for two years before all the updates were complete.
And it wasn’t always plain sailing, as Kroenert explained. “We had to be willing to let some things break sometimes. We’d do something, test it, push it out - and it looked great - but once you then hit 5500 boxes that have all been built differently, some things might break. But then we’d look at the step forward we’d taken.
“There would be wins along the way. When we used to have this whole heap of stuff to do, now it just happens as a part of compliance - every module we fixed took away one more thing we previously had to do.”
Now, instead of bouncing reports between teams and spending long hours explaining each ream of code with auditors, the enterprise platform team now simply has to present them with a visual traffic light system that automatically flags whether a system is compliant or not.
In addition, Puppet ‘patched’ together ANZ’s operating systems to ensure when an update was carried out in one place, the same update reached the whole ecosystem. The vendor also installed the same interface across all the systems to make the environment more manageable.
Since reinvigorating ANZ’s compliance solutions, Puppet has since been hired to automate other departments across its stack.
“Rather than just saying ‘this is garbage, it doesn’t work’, working with Puppet and spending all the time with them to get the data they need is really worth it,” concluded Kroenert. “It went from horror to a nice place to be."
