The backup bug
One recent example comes in the form of a bug. Oracle databases have a feature that supports hot backups. This allows a database administrator to run a command to facilitate a backup of a live database. It's a handy function that can be run easily: 'ALTER DATABASE BEGIN BACKUP' is the command you need. You can then back up the live database until you issue an 'ALTER DATABASE END BACKUP' command that returns the database to normal operating mode. This makes it very simple for a DBA to take care of live backups of production databases.
The problem is that, due to a coding flaw, issuing the 'BEGIN BACKUP' command causes the SCN for the database instance to increase dramatically, so that the SCN continues to increase at an accelerated rate even after the 'END BACKUP' command is given. Thus, performing a hot backup can increase the SCN value by millions or billions quite quickly -- and that elevated growth continues unabated. In most cases the SCN limits are so far out of reach that the occasional jump in number isn't a cause for concern. Admins are unlikely even to notice the problem.
But when you mix the hot backup bug with large numbers of interconnected databases in a massive Oracle implementation, the combination can result in widespread SCN elevation. Some large Oracle customers have hundreds of database servers running hundreds of Oracle instances interlinked throughout the infrastructure. Each one may be tasked with a core service and a few lesser functions -- but like Kevin Bacon and the rest of Hollywood, nearly all are linked together somehow, through one, two, four, or more intermediary servers.
With all of these servers interconnected, their SCNs become synchronized at one point or another. Collectively they might be pushing more than 16,384 commits per second, but certainly not since 01/01/1988, so the SCN soft limit isn't problematic.
But what if a DBA on one part of this Oracle network runs the flawed hot backup routine? Suddenly, the SCN on his Oracle instances shoot up by, say, 700 million, and this number soon becomes the new SCN for all interlinked Oracle instances throughout the organization. Some time later, another hot backup command is issued by a DBA on the other side of the company. The SCN shoots up a few hundred million this time, also synchronized across all connected instances over time.
With the issuance of a few commands, the SCN of the entire Oracle database infrastructure has increased by hundreds of millions or even hundreds of billions in a short period. Even database instances that connect only occasionally, or for a weekly or monthly batch run, might see their SCN numbers jump by trillions.
In such a scenario, it may be only a matter of time before enough backup commands are run to cause the SCN to eclipse the soft limit -- at which point every interlinked Oracle database server suffers some significant problem, refuses connections from other servers, or simply crashes.
Oracle released a patch to fix the arbitrary SCN growth rate bug in the hot backup code before InfoWorld began researching this story. The backup bug is listed as 12371955: "High SCN growth rate from ALTER DATABASE BEGIN BACKUP in 11g." If you have not already done so, Oracle recommends that you install this patch immediately.
But the risk of incrementing the SCN via the backup bug is not the only cause for concern. Perhaps the most important part of our finding is that the SCN can be incremented by anyone who can issue commands on an interconnected database.
Say there's a low-privilege reporting database system that has a read-only database link to a high-privilege database somewhere down the line. All someone has to do is issue a few administrative commands on that low-privilege database to raise the SCN value by, for example, 1 trillion; the next time the high-privilege database receives a connection from that box, its SCN increases to the higher level. It's simple enough to verify the current SCN value, so running a command to push it a few ticks shy of the soft limit is trivial.
Using this technique, a bad actor could potentially cause a systemwide Oracle database communications failure, a shutdown, or a crash with only a few commands on a sideline server -- or even craft a bit of code to mimic a connecting instance. It may be possible, though more challenging, to cause the same problems though a SQL injection attack on a vulnerable application.
Our discovery of this attack method and our communication of this information to Oracle resulted in Oracle's request that InfoWorld hold this story until today, when a patch could be made available. In addition to the method we discovered, we learned of two other, similar means of incrementing the SCN by searching the open Web, although these required higher privilege levels. The patch blocks all three methods that we are aware of, though there may be others.