By Jeff Garbus, CEO, Soaring Eagle Consulting Inc
Cleanup in Aisle 98 – ensuring data consistency in the medical services field
In an economy, which has been inching along, there’s been one industry that’s still exploding in growth and spending ‘the medical field.’
We’re finding that the purchase of smaller, generally also rapidly growing companies, has identified a risk area. When a company is looking to be acquired, it will often cut expenses as far as possible, in order to look good for investors. Many small companies in general start out with an IT staff that is composed of a network administrator who knows how to install a database, who becomes the de facto DBA for the group.
One problem – is somebody caring for the data?
When we bring in a new managed services customer, the first thing we do is an assessment.
Our first question is always, “what does preventive maintenance look like?”
In this case, they thought they had a backup solution, but no other preventive maintenance was being performed.
The first thing we did was make sure backups were all working properly.
The second thing we did was implement preventive maintenance on the servers as quickly as resources would allow.
This is a good thing; we found corrupted data on 3 different servers. We were fortunate that none of this had snowballed into corruption that was irreversible, but it is likely that many of the reports from those servers were returning incorrect or incomplete data. In addition, tasks like index rebuilds were simply not happening. This causes snowballing poor update performance, inconsistent to spiraling bad read performance, as well as massive storage and I/O inefficiencies.
This is the problem with having a network administrator manage databases.
We fixed the data, and now have preventive maintenance scripts running everywhere.
Still in acquisition mode
As additional companies are being acquired, company sizes are on the increase, as is the scale of the data. Once the environment was stabilized, we were given early access to a new acquisition.
The new company, rather than having multiple servers, put all of their data onto a single server into a huge database. At 3T on a MS SQL Server instance, they found that while they wanted to run the standard maintenance, they were unable to perform it within their maintenance window, and had simply given up.
Well, at least they had their backup approach working.
As you start to scale, you need to think outside the box. Some DBMS’ has a set of default maintenance scripts that you can run, but they’re really targeted towards the entry-level DBA who is trying to set up maintenance for a smaller-scale environment.
It’s essential to understand what the maintenance is for so that it can be done piecemeal or in parallel.
Our first step there was to create a high-availability environment. In that technology, you can create an environment where small problems are self-correcting. Then, we create the piecemeal approach to making sure everything is properly maintained.
Don’t overlook the basics. If you don’t know what the basics are, then ask an expert. Ask your collocation provider for current industry stats, but an exceptionally high percentage of businesses (60-70%) shut down when they have catastrophic data loss.
If scale is a problem, ask an expert.
Most scale problems have already been solved in most technologies. Of course, sometimes the solution is to move to HADOOP or in-memory databases or you-name-it depending on what you’re having a problem scaling, but you no longer have to be the one on the bleeding edge.
It’s been stated in print on many occasions that the concept “All problems have solutions” is naïve or immature.
In the database arena, we find that not to be the case. Problems do have solutions.