OraClever

IT Strategies from Oracle

2011-04-11T14:57:00.001+02:00

Having an Enterprise Architecture is essential for successfully rolling out an IT strategy for almost any company. Enterprise Architecture is all about defining what your business wants to do from the perspective of why you want to do it, and translating it into how to do it. There are many frameworks available that help you define an architecture, be it Business, Data, Application or Technology related. For the latter, Oracle has developed a technology architecture, called the Oracle Reference Architecture (ORA). The ORA is part of the IT Strategies from Oracle.

The IT Strategies from Oracle give you a whole library of whitepapers, not only to develop a Reference Architecture for your own, by adapting the ORA to your needs, but it also focuses on the surrounding Enterprise Technology Strategies and Enterprise Solution Designs. In other words, ITSO covers both the horizontal technology perspectives (SOA, BPM, EDA, etc.), but also the vertical business perspectives (Utilities, Government, etc.).

Now, in case you think that this is all about Oracle Technology products, you might be surprised: the whole ITSO / Oracle Reference Architecture is Vendor-Neutral. It is only scoped to Oracle's product portfolio. Now, one might ask themselves: what technology product area's doesn't Oracle have products for, so that shouldn't be too much of an issue as far as completeness is concerned.

ITSO can help you organizing complex product landscapes, by means of a holistic approach to technology adoption. By covering the technology as a whole, you can reduce risk and become more in control of your IT solutions.

The ITSO, and, being part of it, the Oracle Reference Architecture is not an Architecture Framework. For this, many solutions are already available, of which TOGAF and Oracle's Enterprise Architecture Framework (OEAF) are good examples. The ORA can be perfectly integrated in any of the currently available frameworks.

If you are interested in the ITSO, please visit www.oracle.com/goto/itstrategies, you will find a great deal of information there:

Oracle Reference Architectures
These cover

Application Infrastructure Foundation,
Service Orientation,
Service-Oriented Integration,
Software Engineering,
Security,
Management and Monitoring

Enterprise Technology Strategies for Service-Oriented Architecture
Practitioner Guides

SOA Foundation
Creating an SOA Roadmap,
Frameworks for Governance,
Identifying Services,
Determining ROI,
Software Engineering

Enterprise Technology Strategies for Business Process Management
These documents contain

BPM Foundation
BPM Infrastructure
Practitioner Guide on Creating a BPM Roadmap

I am sure you will find something useful in this library for your enterprise information technology strategy.
Be sure to check it out!

Finally took the step: Twitter

2011-02-25T09:28:00.000+01:00

I have just tweeted for the very first time in my life. Awkward feeling, especially since I am not altogether convinced whether twitter really suits me. On the other hand, blogging takes a lot of time which I can spare really, so maybe twitter will fill the gap...
My account, for anyone interested in following me: OraClever (I have no followers yet, so please...?:-)

TOGAF 9 Certified

2011-01-31T14:58:00.000+01:00

In the week before Christmas, I attended a TOGAF 9 course (http://www.the-unit.nl/). This was particularly interesting, since I have certain architectural responsibilities at the company I work for. TOGAF is becoming more and more a world leading framework for developing architectures.
It was a five-day training, and it has been a most interesting week. The trainer (Rienk de Kok) was very knowledgeable at the subject and clearly had a lot of working experience. He led us through the wondrous world of developing architectures, and at the end of the course week, we had to take the first exam (TOGAF Foundation). We all passed the exam, which gave us the key to the next exam: TOGAF Certified. In order to pass this exam, you have not only to know the terms and put them in the right place in the framework (Foundation exam), but also need to know how to apply the framework in a series of 8 cases that are examined in the Certified exam. I must say, it was a difficult exam. Not only because the cases are quite extensive (lots of reading), but the answers to the question that comes with each case each are differently awarded. In other words: 3 of the 4 answers are more or less correct in one way or another. There is one best answer, a second best answer and a least good answer. Beside these 3 answers there is one distractor (if you can call it a distractor... sometimes it is quite hard to tell...).
Today I passed the TOGAF Certified exam, which makes me TOGAF 9 Certified.

5 stars for the training and the trainer!
(and for me, passing both exams at the first attempt :-))

It has been some time...

2010-11-12T20:15:00.000+01:00

...since I last posted here... It has been a busy time. Hopefully I will get back to blogging shortly.
Just be a little more patient...

Oracle Contest: Are You The Smartest 2010 - WINNER!!!

2010-06-10T08:29:00.001+02:00

In the past half year Oracle BeNeLux organized a contest, called Are You The Smartest. This is a contest for Oracle Partners, and covered 7 evening sessions.
The idea behind this contest is to get Oracle Partner consultants up to speed with new products and strategies on the technology level. Seven evenings packed with information on the following topics:

Oracle Technology Overview
Layering your Enterprise Security
Database 11g r2 New Features
Enterprise 2.0 and moreLarge Databases, Partitioning, Exadata, OWB-ODI, Golden GateOracle
Enterprise Manager from OS to Application
Application Grid 11g with WLS, Coherence, Tuxedo, JRockit, RealTime, ...

Every topic was followed by an exam, to be taken right before the next session.
These were no easy exams, I can tell you. 20 questions in half an hour. Half of the questions were taken directly from the session content, the other half from other documentation and information on the web. Scoring is quite particular: every correct answer is a point, every wrong answer is a negative... So in theory, a negative score can be achieved... Hence, study was necessary.

My company (Inter Access B.V.) subscribed with approximately 25 consultants, so we were a "big" partner. One of us was to be rewarded the grand prize: An All-inclusive trip to San Francisco and Las Vegas, including a visit to Oracle Headquarters in Redwood Shores.

Three weeks ago, I received the good news: I am one of the winners of the grand prize! Yesterday evening I was awarded the prize, and next week I will be flying to San Francisco for the trip.

For a complete coverage of the trip, please visit http://ayts2010.blogspot.com/. Stay Tuned!

ACFS: NOT for Database Files!

2010-01-28T20:18:00.004+01:00

ASM Cluster File System was introduced with Grid Infrastructure along with Oracle 11g Release 2. To be honest, I assumed this file system layer was also provided to store database files, a kind of improvement, because ASM, being a black box to many, 'hided' these database files from sight on the Operating System. Which - on the other side - was kind of neat, because chances were lowered these files to be accidentally removed by non-knowledgable users.

It is technically possible to store your database files on ACFS, no question about that. I already deployed a number of test databases on ACFS. However, the Oracle 11g Release 2 Storage Administrator's Guide, Chapter 5 states:

Oracle ASM is the preferred storage manager for all database files. It has been specifically designed and optimized to provide the best performance for database file types.
Oracle ACFS is the preferred file manager for non-database files. It is optimized for general purpose files.
Oracle ACFS does not support any file that can be directly stored in Oracle ASM.

Not supported means Oracle Support Services will not take calls and development will not fix bugs associated with storing unsupported file types in Oracle ACFS.

Especially that last sentence says it all: If you run into trouble with your database stored on ACFS, you will not be assisted by Oracle Support.

Here is the list of file types that are NOT supported to be stored on ACFS:

Control files

Data files

Redo log files

Archive log files

Temporary files

Data file backup pieces

Data file incremental backup pieces

Archive log backup piece

Data file copy

Persistent initialization parameter file (SPFILE)

Flashback logs

Change tracking file

Data Pump dumpset

Automatically generated control file backup

Cross-platform transportable data files

Flash file

Oracle ASM Persistent initialization parameter file (SPFILE)

backup

Oracle Cluster Registry file

Be aware of this when you are designing your Grid Infrastructure!

UKOUG RAC & HA SIG Meeting Agenda Available

2010-01-19T21:58:00.004+01:00

On February 10 I will present at the UKOUG RAC & HA SIG meeting at Oracle in Reading.
The agenda has been published and will contain the following interesting sessions:

10:00
Welcome and Introduction

10:15
Support Roundup - Phil Davies, Oracle

10:35
Recovery of very Large Database in shortest time possible - Arnoud Roth, Inter Access

11:45
ASM Split Brain - Problem Description - Pavel Rabel, DSP Managed Services

12:15
Application Consolidation Experiences on a 3-node RAC Cluster - Martin Bach, Markit Group Ltd.

14:00
Using RAC and Grid for Database Consolidation - Panel Discussion

14:45
Rolling Database Upgrade, a Real World Experience - Thomas Ramanauskas, ekc

15:15
Oracle RAC on Cisco UCS hardware, boxes to blades by brunch - Dev Nayak, DSP Managed Services and Carl Bradshaw, Cisco

16:00
Stretched RAC Cluster with 11gR2, David Burnham, Ioko365

16:30
Close

Looks to be a promising day!
I am really looking forward being there.

Happy Holidays!

2009-12-24T23:33:00.002+01:00

Wishing everybody out there a very merry Christmas and a prosperous 2010, full of learning and sharing!
Keep in Touch!
Arnoud

Presenting @ UKOUG RAC & HA SIG meeting on Feb 10

2009-12-22T21:23:00.003+01:00

On February 10, 2010, the UKOUG RAC & HA SIG will organize it's half-yearly SIG meeting.

I am very pleased to tell you that I have been invited to join this meeting and present about the recovery method I have been exploring lately (see this post, with all its referring posts).

The agenda is not yet fully confirmed, but there will be

a forum discussion on the topic of using Grid and/or Real Applications Cluster for consolidation;
Martin Bach (Oracle 10g Certified Master) will do a presentation.
My presentation - Lightning fast recovery of (physically) corrupted databases without the need to restore the backup - or - To Restore Or Not?

Other presentations / sessions have to be confirmed yet, but it all seems quite promising (if I may speak - partially - for myself:-).
Once the program is confirmed completely, I will post about it again.

The event will take place on Feb. 10 at the Customer Visitor Center at Oracle Thames Valley Park in Reading, UK.

Looking forward to an incredible day of learning and sharing.

E-Business Suite Release 12.1.2 available

2009-12-22T13:17:00.004+01:00

Having been ill for a couple of days, I missed Steven Chan's announcement for Oracle E-Business Suite Release 12.1.2.
In an earlier post I already elaborated on the new Technology Features of this release.
I will get my 12.1.1 environment started up this weekend and see whether I can upgrade to 12.1.2 according to Metalink Note 845809.1.

Nice new feature: Oracle Database 11gR2 is certified with R12.1.2 (or should it be the other way around?)!

I will keep you updated on my progress.

E-Business Suite R12: Shared Application Tier File System

2009-12-09T13:59:00.003+01:00

In Oracle E-Business Suite Release 12, one of the eye-catching new features was the Instance Home, or $INST_TOP. The result of this Instance Home is that you can now share the APPL_TOP between multiple installations of Oracle E-Business Suite.
I have seen a number of articles and discussions on the Internet discussing the reasons why one would want to share the APPL_TOP between E-Business Suites.

Why share?
Personally, I didn't see any reason why I would want this, other than to see it work and be able to use the feature, when time would come that I would need it.
I cannot remember where anymore, but I saw a reason lately that actually made sense to me:
As a DBA, it could be of use when I would want to replay issues that have been reported to me, being sure that I would be using the exact same APPL_TOP. I could set up a separate E-Business Suite, sharing the APPL_TOP, but having my own database. Now I can play around with my own E-Business Suite, without making changes to the production data, but I am still using the very same code as in the production environment. To make sure I do not break anything, I could even mount the APPL_TOP read only.
Recently I found another reason why it could be interesting to share the APPL_TOP: When your production environment consists of multiple E-Business Suites. I heard about Oracle's recommendations to separate Advanced Supply Chain Planning from your OLTP E-Business Suite environment. You would have to maintain two E-Business Suites, and I can see a couple of reasons why you would want to have identical code versions on each of them. In order to keep the two (or any number of) environments synchronous, you might want to implement the shared APPL_TOP.

Share what? APPL_TOP vs. Application Tier File System
By the way, can "shared APPL_TOP" also be read as "Shared Application Tier File System"? I have taken up the habit of asking whether APPL_TOP means Application Tier File System, or it just reflects the $APPL_TOP directory with all its subdirectories. It happens to be that the two are not the same, even though many of us use the two simultaneously.
In my opinion (but that may be because my experience goes back to the days these technologies were introduced), the APPL_TOP is just the ($)APPL_TOP: the top level directory that contains the product tops (/whatever_directory/apps/apps_st/appl).
The Application Tier File System is more than just the APPL_TOP. It contains the whole set of code that makes the Application Tier, i.e. the APPL_TOP, COMMON_TOP, but also the 10.1.2 and 10.1.3 ORACLE_HOMEs that reside under /whatever_directory/apps/tech_st. To conclude: The Application Tier File System is everything that resides under /whatever_directory/apps. That is a whole lot more.
But hang on? The question now is what Oracle means by stating that the APPL_TOP can be shared among multiple E-Business Suite environments. Do they mean just the APPL_TOP, or the Application Tier File System?
The official answer can be found in Metalink Note 384248.1 , Appendix B where you can read that it is possible to share the complete Application Tier File System among multiple unshared E-Business Suite Databases.

"...Coolest Recovery Demo Ever..."

2009-11-04T14:12:00.003+01:00

Yesterday, I got the chance to perform a presentation at the Dutch Oracle User Group (OGh) DBA day about a topic that I have blogged about a couple of days ago: Database Crash Recovery : To Restore Or Not Restore.

I proved my "theory" with a live demo. In VMware I crashed my database after having it backed up, after which I was able to recover within 5 minutes, without restoring the backup. All changes were kept, even the changes after the last incremental backup was taken... (I admit, I was impressed myself, too!:-)

The presentation went well, and today I received some comments from some of the attendees, telling me that the presentation was received extremely well. One of the attendees even twittered about "the coolest recovery demo ever".

For anyone interested in the solution/approach: The concepts of this approach are outlined in this and this article, founde elsewhere on my weblog.

Happy reading and testing!

Oracle E-Business Suite Release 12.1.2 - New Features

2009-10-28T11:04:00.007+01:00

Wondering what the next release of E-Business Suite R12 will contain? Steven Chan posted a weblog about the availability of the 12.1.2 RCD yesterday, way before R12.1.2 is made available (the release date is not yet known).
However, it is always interesting to see what the next upcoming release will contain. I will stick to the Applications Technology, because that is my part of the pond:

Technology Highlights

The most eye-catching new feature is support for 11gR2 database. Unfortunately (in a way) 11gR2 will not be shipped with 12.1.2. It'll still come with 11.1.0.7.
On the middle-tier, support is introduced for AS 10gR3 (10.1.3.5) which will be shipped with the release, JDeveloper 10.1.3 and it uses the Sun Java Plug-in version 6.
The Application Server for Forms and Reports will be 10.1.2.3. The maintenance pack will either include or require customers to upgrade to this version.
Of all externally installed Oracle technology (WebCenter, Discoverer, Portal, Internet Directory, SSO, BPEL PM, OBIEE, OEM, etc) the most recent version will be certified with 12.1.2.

Internationalization Highlights

Added support for Lithuanian, Indonesian and Ukrainian languages.
Space and single quote number group separators
First day of the week support (choose whichever day is the first day of the week)
The Gantt chart and inline date picker features support Thai and Hijrah calendars

Oracle Applications Framework Highlights

Redesigned Home Page
new folder-like structure, better spacing, favorites are now in a pull-down menu. In general the home page is more compact and allows for easy extension and custom content.
New Navigator Menu and favorites Pull-Down menue in the Global Header area
The Navigator Menu now appears in every OAF page (Pull-down menu). Favorites menu is also a pull down menu from the Global Header area and allows for marking the current page as favorite.
Look-ahead capability in lists of values
When entering a limited number of characters in a field, an inline window will let you select from a list of values based on the characters you typed. This way, you can limit the number of values in the inline window and ease your search for the correct value.
ability to add, view, edit and delete attachments inline
Inline attachments now have a hover-over interaction allowing users to view, add, edit and delete attachments to an entity. This capability eliminates the need to go to the Add and View attachments pages to maintain attachments.
pop-up component
This allows applications to display a pop-up window on top of an OAF page.
Portlet Generator
For static conversion of a standalone OAF region into a portlet. These can be embedded in any portal application or WebCenter.
A new component: rich container that can hold OBIEE content.
Through personalization, users can configure this component to hold a certain OBIEE report that accepts specific parameters.

Oracle Application Object Library Highlights

Oracle Access Manager integration support
OAM can be used in stead of Oracle Single Sign On for thos applications or products that support it, but it can also be used together with OSSO to provide centralized authentication.
AOL Java APIs
Support for Oracle E-Business Suite Release 12.1.2 is provided
Scheduling Charts UI Enhancements
Dragging task numbers into the scheduling chart to create new tasks and shifts on the fly. Simplified ability for creating, assigning, searching and displaying new task assignments from a single source.

Diagnostics Framework

Added support for XML file type diagnostics.

Integrated SOA Gateway

SAML Token Support
E-Business Suite environments can trust external systems. These external systems can use a public key to authenticate a SOAP request using SAML.
Enhanced SOA Diagnostics
To check the health of the EBS Integrated SOA Gateway.
New interface type: Java APIs for Forms
these are XML document based interfaces, wrapped in Java classes to execute logic in Oracle Forms. Integrated in the Oracle Integration Repository (iRep)
Integrated SOA Gateway Customization
Support for custom integration interfaces, that can be deployed as custom Web Services. Customization feature is provided through the Integration Repository Parser.

Web ADI Highlights

Introducing the Web Applications Desktop Integrator Extension Framework
wizard-based UI to create and manage desktop integrator solutions, providing a user-friendly approach to create custom integrators, without having to know the complexity of the EBS Schema.

Conclusion

My personal curiosity is focused on 11gR2 (I would be really interested in seeing the E-Business Suite running on 11gR2 Grid Infrastructure!). For sure there is enough to lookout for. Again, I am eagerly awaiting this new release.

using the RMAN backup as copy as base for Disaster Recovery

2009-10-27T09:55:00.001+01:00

In an earlier post I have elaborated on the advantages the backup as copy feature offers to quickly recover from a damaged database.

In this article I will try to show how this advantage goes even further by adding another server to the infrastructure, introducing some kind of Disaster Recovery solution. OK, it is not fully featured like Oracle Dataguard, but it works and can be the first step towards Disaster Recovery. No need for a standby database, redo log shipping etc. It is all plain simple and straight-forward.

Shopping list:

2 servers, not necessarily equal in specifications, but having identical Operating Systems
Clustered File System (OCFS2 will suffice, even NFS will do the job)

Preparations

First of all, you need to set up OCFS. Tons of information out there on the Internet to get your OCFS working. It is not difficult, See this PDF for information.
What I have done to make this work is set up 2 OCFS2 File Systems, one under /oradata and one under /orafra.
Next, install the Oracle RDBMS software on both machines, each using the same location for the Oracle Home and installing identical copies.

Once you have OCFS2 set up on your 2 servers, create or move your database onto /oradata, and make sure the DB_FILE_RECOVERY_DEST parameter points to /orafra and the DB_FILE_CREATION_DEST points to /oradata. Don't forget the DB_FILE_RECOVERY_DEST_SIZE parameter! Set it to a size equal to the file system where it is located, to make sure you will not run out of space too soon.

Your infrastructure will look like this (more or less):

/oradata is the location for you database, /orafra is for the Flash Recovery Area.

Now we can prepare the secondary machine to serve as Failover server.

To do this, take the following steps:

1. Copy the server parameter file and bring it to the secondary machine under $ORACLE_HOME/dbs. Edit this file and make adjustments to the following parameters:
CONTROL_FILES = /orafra/SID/controlfile/cp_cntrl.ctl
DB_FILE_RECOVERY_DEST = /oradata
DB_FILE_CREATE_DEST = /orafra
Optionally, you can create an spfile from this parameter file on the secondary machine.

2. Copy the /etc/oratab file to the secondary node

3. Create the necessary dump directories ($ORACLE_HOME/admin)

Now you are ready to go! Bring up your database on the primary machine and create a full backup as copy from your database:

allocate channel for maintenance type disk;
configure controlfile autobackup on;
configure default device type to disk;
run
{
RECOVER COPY OF DATABASE WITH TAG ‘IMG_COPY’ UNTIL TIME ‘SYSDATE – 1’;
BACKUP INCREMENTAL LEVEL 1 FOR RECOVER OF COPY WITH TAG ‘IMG_COPY’ DATABASE PLUS ARCHIVELOG DELETE INPUT;
copy current controlfile to ‘/orafra//controlfile/cp_cntrl.ctl’;
}

This script will do everything for you. See this post for an explanation of the script.

Now let's make some changes to the database and run the backup script again (NOTE: in order to have all changes you made to the database immediately reflected in the copy of your database, remove the UNTIL clause in the first line of the backup script).

What you can do now, is to simulate a crash of the database (there are a lot of ways to do this, I would recommend you just issue a shutdown abort at your first attempt).

Go to the second machine, log in as the oracle user, source your environment and perform the following steps to recover your database:

start an rman session to the database (which is not started yet)
mount the controlfile (located in /orafra/SID/controlfile, pointed to by the parameter file)
issue "switch database to copy"
issue "recover database"
issue "alter database open resetlogs"

That is all it takes. Now your database is opened and available from the secondary machine. Now make sure that you reverse file locations in the backup scripts on the secondary machine to reflect the database file locations and the Flash Recovery Area. The database files should be located in /orafra, the flash recovery area should be located in /oradata. If you want to switch back, just use the backup script to create another backup as copy and switch back the way you did earlier. Make sure you have defined the correct file locations on the primary server, alike the secondary machine, to smoothen the switch.

Caveats

Like what is the case with any other regular backup, when you have made changes to the physical structure of your database (like adding datafiles, tablespaces, etc), an incremental backup needs to be taken immediately after, to prevent issues while recovering the database from the copy.

Another caveat is that it is necessary to have both database servers available in your tns alias being used to access the database. In such a case it is important to have load balancing set to off and failover set to on, having the primary as the first address.

Alternatives

If your server is still available, but the database has become corrupted, you can easily switch to the copy without the need for the secondary machine with a similar procedure as mentioned above. This way you will always have the choice of what to do.

Happy testing!

Oracle 11g Release 2 Clusterware New Feature: Policy-based Cluster and capacity management

2009-09-23T13:13:00.004+02:00

I was reading through the Oracle 11g Release 2 Real Applications Cluster New Features List and stumbled upon this really cool new feature. I have to tell you about it, because this, imho, is one of the things that makes Oracle Clusterware a “Grid”.

So, what is Policy-based management? In order to answer that, you must first understand some basic changes in 11gR2. This is what the manual says:

With Oracle Clusterware 11g release 2 (11.2) and later, resources are contained in logical groups of servers called server pools. Resources are hosted on a shared infrastructure and are isolated with respect to their resource consumption by policies, behaving as if they were deployed in a single-system environment.

What does this mean? I have a bunch of resources and a bunch of servers (a grid) and subsets of these servers are combined in server pools. The resources are assigned to these server pools, but the server pool acts as if it were just one system. The isolation is done on a policy base. The whole idea is being able to limit the capacity that is assigned to a resource, and have a set of server pools to service those resources. BTW. It is possible to assign a server to multiple server pools.

This kind of management enables you to dynamically assign capacity to meet your policies and priorities, as well as to allocate (physical) resources by importance. Your critical applications will always have a minimum of resources at its disposal. It also gives you the opportunity to limit applications to use a set of servers.

How does it work? You just assign the minimum and maximum number of nodes to a resource, as well as the priority and Oracle Clusterware will do the rest.

The servers will get a number of attributes, like a name (basically the node name), the pools to which it belongs and the state it is in with some details.

How do Server Pools work? They are sets of servers, that service resources. What they will try to do is to spread the workload over the available servers in the pool. It is possible to limit a database to run in a specific server pool.

The top-level server pool is an exclusive server pool, which means that any given server can only be a part of one top-level server pool. It is impossible to be a member of multiple top-level server pools. This way, the grid or cluster is logically divided.

All servers that join the cluster/grid are automatically assigned to the Free Pool initially. This Free Pool is a server pool that is automatically created at installation. From here, servers can be assigned to other server pools. Another pool, the Generic Pool is also automatically generated, and all servers will be part of this pool too, in order to ensure compatibility with prior releases. Any pre-11gR2 database will need to run in this generic server pool.

The server pool is given an minimum and maximum number of nodes. As soon as the number of nodes decreases below the minimum value, Clusterware will automatically take necessary action and take servers from other pools to increase the number of servers to meet the required minimum number.

If this poses a problem for other server pools, decisions are made upon the importance of the pools. When importance is equal, it is only possible to take servers, if the number of servers is higher than the minimum value.

All of the above can be done completely dynamically, however, manual manipulation is still possible. You might think: "Where is my control?", but if you set it up carefully, it could save you a bundle. For example, when you configure Grid Plug and Play (another one of those new features), it is possible to have a database bring up a new instance in the server pool and automatically create new log threads, undo tablespaces, etc. Isn't that way cool?

Oracle 11g Release 2 - ADVM (ASM Dynamic Volume Manager)

2009-09-23T09:00:00.004+02:00

One of the new features in Oracle 11g Release 2 is Automatic Storage Management Cluster File System. This file system provides concurrent access from multiple (clustered) nodes.

ACFS is provided through the Automatic Storage Management Dynamic Volume Manager. In order to have a file system, one needs a volume (be it physical or logical) to put it on. Oracle has developed a Dynamic Volume Manager (ADVM) that can provide these volumes from ASM Diskgroups. You can use ADVM to create volumes that are replicated within the ASM diskgroup mechanism – by means of normal or high redundancy. Multiple volumes can reside on one diskgroup. The ADVM provides a device driver interface to the ACFS client.

Due to the mirroring features of ASM, it is possible to have a mirrored file system (even across a stretched cluster, when your storage infrastructure allows for such an architecture) that will leverage the features of ASM to provide consistency over hardware failure and ASM instance crashes.

You can use the file systems provided through ADVM not only for database files, but also for regular files such as executables and log files. During creation of the file system you have the choice to designate the file system for Oracle Home functionality. By choosing this option, the file system is registered in the Cluster Registry as a managed resource, giving Oracle the opportunity to layer dependencies on this file system for the use of Oracle Homes.

Because of the fact that Oracle ADVM Volumes are technically spoken ASM files located on ASM Disk groups, and the fact that the Dynamic Volumes do not use the traditional device partitioning, it enables Oracle to extend some of the ASM features to the ASM Clustered File Systems, which are created inside these ADVM Volumes, such as dynamic resizing or dynamically adding volumes. This makes ADVM and ACFS a far more flexible solution than traditional physical devices.

Important Note: Oracle ADVM supports all storage solutions supported for Oracle ASM with the exception of NFS and Exadata storage. That will have to wait for some time still…

The above picture shows the User Interface, provided through the ASM Configuration Assistant (asmca), to create file systems inside ASM using the ADVM.

Oracle ACFS advanced topics

2009-09-16T11:41:00.006+02:00

While everybody is exploring new features of the latest release of Oracle 11g, we're all very excited about all the new possibilities.
One of those new features is Oracle ACFS (ASM Cluster File System), wich provides an Oracle Home shared file system or a general purpose clustered file system on ASM storage. These are very nice features that will be used by many customers, in time to come. My first experiences are quite good. However, it is of great importance to know the limitations of those new features as well. Oracle has - on the topic of ACFS - thought about this very carefully, by adding an Appendix to the Storage Administrator's Guide, called ACFS Advanced Topics. Amongst these, a number of limitations are discussed. Well, one can argue whether these are actual limitations, that really limit you in operating ACFS, but it is good to know about these when planning for you Grid Infrastructure.

Oracle ACFS Disk Space Usage
Some Facts and Figures: Oracle ACFS supports:

64 million files in a file system,
63 snapshots,
64 mounts on 32 bit systems,
256 mounts on 64 bit systems.

Oracle allocates metadata for each node that mounts the file system, comprising approximately 64 – 128 MB per node. Also, some space is reserved for local bitmaps to reduce contention on the global bitmap in order to find free space. This can add up to 128MB per node. All of this space is reported as “in use” when querying the file system for disk space.

Oracle ACFS Error Handling
CAUTION: It is very important to unmount any file system that is using the Oracle ASM Dynamic Volume Manager (ADVM), before shutting down an Oracle ASM Instance. Failing to do so can result in I/O failures. One can never guarantee that the ASM instance(s) will never fail, but in case it does, volumes must be closed and opened again to be able to access them. Dismount the file systems that were mounted during failure and after the instance is restarted, mount the corresponding disk group with the volume enabled, after which you can mount the file system again.

Therefore, if any file systems are currently mounted on Oracle ADVM volume files, the SHUTDOWN ABORT command should not be used to terminate the Oracle ASM instance without first dismounting those file systems.

When a metadata write fails, regardless whether it be an ASM or storage failure, ACFS isolates the errors to a single file system and puts it in an off-line state. The only thing you can do with it is dismount. Other nodes (when available) recover the transaction, if it can write to that particular storage. After this, the file system can be mounted again.

When the file system being put offline is in use by processes or users, it might not be possible to unmount it. The administrator has to identify the processes that are claiming the file system and stop them before unmounting.

Oracle ACFS and NFS
Because block device major numbers of ADVM devices can be different after a reboot, the -fsid=# should be used when exporting NFS file systems. The option forces the clients to use this number (#) when communicating with this file system. Obviously, the number should be unique across the cluster. It replaces the number derived from the major/minor number of the device itself, since it can change with reboots.

Limits of Oracle ADVM
Some Facts: The default configuration for an Oracle ADVM volume is four columns of 64 MB extents in length and a 128 KB stripe width. Oracle ADVM writes data as 128 KB stripe chunks in round robin fashion to each column and fills a stripe set of four 64 MB extents with 2000 stripe chunks before moving to a second stripe set of four 64 MB extents for volumes greater than 256 megabytes. Note that setting the number of columns on an Oracle ADVM dynamic volume to 1 effectively turns off striping for the Oracle ADVM volume (are you still with me?;-)

On Linux platforms Oracle ASM Dynamic Volume Manager (Oracle ADVM) volume devices are created as block devices regardless of the configuration of the underlying storage in the Oracle ASM disk group. Do not use raw (8) to map Oracle ADVM volume block devices into raw volume devices.

Oracle ACFS and Oracle Restart
Oracle Restart does not support root-based Oracle ACFS resources for this release. As a result, the following operations are not automatically performed:

Loading Oracle ACFS drivers
Mounting Oracle ACFS file systems listed in the Oracle ACFS mount registry
Mounting resource-based Oracle ACFS database home file systems

The Oracle ACFS resources associated with these actions are not created for Oracle Restart configurations.
In short, the following resources are only supported for the Oracle Grid Infrastructure Cluster configurations, and are not supported for Oracle Restart Configurations:

The Oracle ACFS drivers resource (ora.drivers.acfs),
The Oracle ACFS registry resource (ora.registry.acfs)
The Oracle ACFS individual file system resource ora.diskgroup.volume.acfs)

Source: Oracle Database Storage Administrator's Guide, 11g Release 2, Appendix B

ASM Cluster File System: Available Storage From the ASM Perspective

2009-09-15T08:07:00.005+02:00

I was checking my system this morning, having it booted for the first time after the installation of Oracle 11g Release 2 Grid Infrastructure.
I opened up SQL*plus for a connection to ASM, to see what the disk space usage was.
I found the following information:

The first section of the screen shows the total amount of diskspace and available diskspace from an ASM perspective, the lower part of the screen shows the file system usage.
Here you can see that the file system, even though it has practically all space available, has claimed all of its space in ASM. Logically, you might think, but since the two are so tightly integrated, might surprise a bit.
The principle is the same compared to 10g ASM with Oracle Datafiles. The datafile itself might not been occupied fully, still all of the space that the datafile is configured with needs to be allocated in ASM. When you want to see what your actual usage is on an ASM disk and how much you can grow, you would need to calculate the free space in your file system and add the FREE_MB column from within ASM to it to determine the exact free space on your ASM Disk(s).

Configuring ASM Clustered File Systems (ACFS) in Oracle 11g Release 2

2009-09-11T16:23:00.010+02:00

After having finished the installation of Oracle 11g Release 2 Grid Infrastructure, the next step is to configure your storage.
ASM has always been a layer between the hardware and the database files.
Starting with Oracle 11g Release 2, Oracle has introduced a second cluster file system (next to OCFS2) called ACFS (ASM Cluster File System), to be hosted within ASM This is a major new feature of Oracle 11g Release 2.
This cluster file system not only supports binaries, like the ORACLE_HOME. In fact, what we'll see further on in this post, there is a differentiation between a file system for Oracle Home purposes and for General Purposes.
In order to setup an ASM Cluster File System, you simply start the ASM Configuration Assistant. Source the ORACLE HOME of Grid Infrastructure, make the bin subdir accessible from the PATH variable and simply type: asmca
What you will get is something similar to this (click on the images to get a larger view):

You can see the four tabs, ASM Instances, Disk Groups, Volumes and ASM Cluster File System. We'll cover each of them later on. For now, only the first two tabs are shown.

What we'll have to do first is to create a diskgroup. I happened to be prepared for this, so I created an ASM disk prior to running this Configuration Assistant. Just hit the "Create" button and you'll get:

I have called my first disk group DATA. By clicking on the button "Show Advanced Options" you will see the compatibility parameters that can be set per diskgroup (ASM, Database and ADVM (ASM Dynamic Volume Manager) compatibility). Select the disk(s) and click on OK to create the Disk Group:

Well, I think that was a success:-)
Next, go to the Volumes tab. We will have to create a volume first. By the way, doesn't this sound like an old-fashioned Volume Manager setup? It does, because it actually is. The difference with all the OS's proprietary volume managers is that this one is optimized for Oracle, and that should be enough to prefer this one over anything else.

Hit Create and this is what you will get:

Now, be a little careful here. Notwithstanding the fact that the CA tells you that (in this case) you have 14,91 GB of space, you cannot use it all. I have tried 14,91, 14,9, they all failed. The example above shows 14.5, which worked for me. Don't take all you can get, don't get too greedy:-). You have to reserve some space for management, apparently.

Another success!
Now we're almost there, and the most interesting part has to come yet: The ASM Cluster File System tab:

Again, just hit the Create button to get started:
Interesting choice here: You can choose between a Database Home File System or a General Purpose File System. When you choose your file system to be a database home file system, it will be registered under Grid Infrastructure as a managed resource, so Grid Infrastructure can maintain dependencies (- hey, we can all read, you know...). Anyway, I chose the Database Home File System for DATAVOL01. Just enter the mountpoint (/ora11g in the example), who should own it and to which group it should belong:

There is a button there at the bottom: "Show Command", which is worthy to be investigated. It shows you the commands that will be executed to create the ACFS and the registration command:

And the bottom half of the commands:

After the file system has been created, it needs to be registered, by running a script. This needs to be done by a privileged user (e.g. root):

Screenshot of running the script, and Oh my! It's there! I just created a clustered file system under ASM and it is registered in Grid Infrastructure:

OK. Now we'll configure another file system, this time a General Purpose File System. Starting with the second tab in the Configuration Assistant (Disk Groups), define the name of the disk group (DATABASE). You can see that suddenly I have got another set of disks available. Well I created those earlier, but didn't register yet them with ASMlib, while going through the previous cycle.
This time I have got two disks, and I will both use them for my DATABASE disk group. Still with External Redundancy, because the diskspace is too costly for me.

After the Disk Group has been created, I create a volume on it:

After that, I come to the ASM Cluster File System tab again and now I choose a General Purpose File System. I will register this one as well, hopefully resulting in an automount feature at boot time.

The file system has been created and registered successfully:

Now you have two clustered file systems, one designated for Oracle Homes, the other for anything you would like (like databases?).
Next step is to install Oracle 11g Release 2 Database software. That will be covered in the next post.
Happy testing!

Networking Requirements for Oracle Grid Infrastructure

2009-09-09T15:44:00.014+02:00

With the release of Oracle 11gR2, Oracle has provided a whole new concept for the Grid Infrastructure. One of the most eye-catching requirements is a new Network Infrastructure requirement. It is no longer possible (maybe with some hacking and tricking) to just have a couple of Network Interface Cards and a bunch of IP addresses configured through the hosts file in /etc. I will try to summarize the requirements that Oracle has defined, and how to implement them:

Network Hardware Requirements

Oracle states the following requirements:

Each node must have at least two network adapters or network interface cards (NICs): one for the public network interface, and one for the private network interface (the interconnect).If you use multiple NICs for the public network or for the private network, the recommendation is to use NIC bonding. Public and private bonds should be separated
The interface names Oracle will use for the network adapters for each network must be the same on all nodes.In other words: The public network adapter should, for example, be called eth0 on all nodes and the private adapter eth1 on all nodes.
The public network adapter must support TCP/IP.
The private network adapter must support the user datagram protocol (UDP)High-speed network adapters and switches that support TCP/IP (minimum requirement 1 Gigabit Ethernet) are required for the private network.

IP Address Requirements
Before starting the installation, you must have at least two interfaces configured on each node: One for the private IP address and one for the public IP address.

Reading the installation manual for Grid Infrastructure (of which btw this article is a summary) a DNS server and a DHCP server are now required to setup an Oracle Grid Infrastructure.

You can configure IP addresses with one of the following options:

Oracle Grid Naming Service (GNS)
Using a static public node address and dynamically (DHCP) allocated IP addresses for the Oracle Clusterware provided VIP addresses. The hostnames will be resolved using a multicast domain name server, which is configured as a part of Oracle Clusterware, within the cluster. If you plan to use GNS, then you are required to have a DHCP service running on the public network for the cluster which will be able to provide an IP address for each node's virtual IP, and 3 IP addresses for the cluster used by the Single Client Access Name (SCAN) for the cluster.

Fixed IP addresses, assigned on a DNS Server for each node.
In other words, you can no longer trick around by defining the hosts in the /etc/hosts file. DNS is a requirement, as well as DHCP. Well, that may not be the case, but I will have to test this still...

IP Address Requirements with Grid Naming Service
If you enable Grid Naming Service (GNS), name resolution for any node in the cluster is delegated to the GNS server, listening on the GNS virtual IP address. You have to define this address in the DNS domain before you start the installation. The DNS must be configured to delegate resolution requests for cluster names (any names in the subdomain delegated to the cluster) to the GNS. When a request is made, GNS processes the requests and responds with the appropriate addresses for the name requested.
In order to use GNS, the DNS administrator must establish a DNS Lookup to direct DNS resolution of an entire subdomain to the cluster. This should be done before you install Oracle Grid Infrastructure! If you enable GNS, then you must also have a DHCP service on the public network that allows the cluster to dynamically allocate the virtual IP addresses as required by the cluster.

IP Address Requirements for Manual Configuration
If you choose not to enable GNS, then the public and virtual IP addresses for each node must be static IP addresses, but they should not be in use before you start the installation. Public and virtual IP addresses must be on the same subnet, but that should be no news, since that has been the case since the days VIPs were introduced (Oracle 10g).
The cluster must have the following IP addresses:

A public IP address for each node
A virtual IP address for each node
A single client access name (SCAN) configured on the domain name server (DNS) for Round Robin resolution to at least one address (three addresses is recommended).

SCAN
The single client access name (SCAN) is a name used to provide service access for clients to the cluster. One of the major benefits of SCAN is that the SCAN enables you to add or remove nodes from the cluster without the need to reconfigure clients This is because the SCAN is associated with the cluster as a whole, rather than to a particular node. Another benefit is location independence for the databases. Client configuration does not have to depend on which nodes are running a particular database. Clients can however continue to access the cluster like with previous releases. The recommendation is to use SCAN.
The SCAN addresses must be on the same subnet as virtual IP addresses and public IP addresses. For high availability and scalability, you should configure the SCAN to use Round Robin resolution to three addresses. The name for the SCAN cannot begin with a numeral. For installation to succeed, the SCAN must resolve to at least one address.
It is best not to use the hosts file to configure SCAN VIP addresses. In stead, use DNS resolution for SCAN VIPs. If you use the hosts file to resolve SCANs, then you will only be able to resolve to one IP address. The result is that you will have only one SCAN address. It could possibly be a workaround for lab circumstances, but again, I didn't test it this way yet.

DNS Configuration for Domain Delegation to Grid Naming Service
If you plan to use GNS, then before grid infrastructure installation, you must configure your DNS server to forward requests for the cluster subdomain to GNS.
In order to establish this, you must use so-called delegation. This is how you configure delegation: In the DNS, create an entry for the GNS virtual IP address. For example:
gns.domain.net: 192.168.146.51 (The address you provide must be routable)
In the DNS, create an entry similar to the following for the delegated domain, where cluster.domain.net is the subdomain you want to delegate:
cluster.domain.net: NS gns.domain.net
You must also add the DNS Servers to the resolv.conf on the nodes in the cluster

Make sure that the nis entry in /etc/nsswitch.conf is at the end of the search list. For example:
hosts: files dns nis

Grid Naming Service Configuration Example
When nodes are added to the cluster, the DHCP server can provide addresses for these nodes dynamically. These addresses are then registered automatically in GNS, and GNS provides delegated resolution within the subdomain to cluster node addresses registered with GNS.
Because allocation and configuration of addresses is performed automatically with GNS, additional configuration is no longer required. Oracle Clusterware provides dynamic network configuration as nodes are added to or removed from the cluster. The following example should make things a bit clear (click on the image to get a larger view):

The above table reflects a two node cluster with GNS configured. The cluster name is "cluster", the GNS parent domain is domain.net, the subdomain is cluster.domain.net, 192.0.2 in the IP addresses represent the cluster public IP address network, and 192.168.0 represents the private IP address subnet.

Manual IP Address Configuration Example
If you decide not to use GNS, then you must configure public, virtual, and private IP addresses before installation. Also, check that the default gateway is reachable.
For example, with a two node cluster where each node has one public and one private interface, and you have defined a SCAN domain address to resolve on your DNS to one of three IP addresses, you might have the configuration shown in the following table for your network interfaces:

You do not need to provide a private name for the interconnect. If you want name resolution for the interconnect, then you can configure private IP names in the hosts file or the DNS. However, Oracle Clusterware assigns interconnect addresses on the interface defined during installation as the private interface (eth1, for example), and to the subnet used for the private subnet.
The addresses to which the SCAN resolves are assigned by Oracle Clusterware, so they are not fixed to a particular node. To enable VIP failover, the configuration shown in the preceding table defines the SCAN addresses and the public and VIP addresses of both nodes on the same subnet, 192.0.2.

Network Interface Configuration Options
An important thing to be aware of is when you use NAS for RAC and this storage is connected through an Ethernet network. In this case, you must have a third network interface for NAS I/O, otherwise you can face serious performance issues.

Enable Name Service Cache Daemon
You should enable nscd (Name Service Cache Daemon), to prevent network failures with RAC databases using Network Attached Storage or NFS mounts.
Use chkconfig --list nscd
To change the configuration, enter one of the following command (as root):
# chkconfig --level 35 nscd on

If you meet either one of the above method requirements (GNS or Manual configuration), you are safe to install Oracle Grid Infrastructure.

Source: Oracle 11gR2 Grid Infrastructure Installation Guide

Oracle 11gR2 Grid Infrastructure Installation

2009-09-04T20:14:00.014+02:00

One of the new "features" of Oracle 11gR2 is the Grid Infrastructure. Oracle 11gR2 Grid Infrastructure is the collection of infrastructural components provided for Oracle Database and other software.
I tested the installation of Oracle Grid Infrastructure on two virtual machines running OEL 5.3, operating under VMware Server 1.08. I suppose it is not supported this way, but it can give an idea of what to expect during the installation. The screenshots below show the installation procedure. I will add some comments when appropriate, especially to emphasize on new features. NOTE: Click on the screenshot to get a larger view.

The first installation screen. The bullet shows which option I chose here.

Usually the Advanced option is for experienced users. Well, at least I think I am experienced, I suppose...

Languages. Anyone ever chose additional languages here? I never did.

Nice new feature here: SCAN (Single Client Access Name): All the servers in the cluster will act upon a single hostname. This way, the cluster becomes completely transparent.

GNS: If you enable Grid Naming Service (GNS), then name resolution requests to the cluster are delegated to the GNS, which is listening on the GNS virtual IP address. You define this address in the DNS domain before installation. The DNS must be configured to delegate resolution requests for cluster names (any names in the subdomain delegated to the cluster) to the GNS. When a request comes to the domain, GNS processes the requests and responds with the appropriate addresses for the name requested.
To use GNS, before installation the DNS administrator must establish DNS Lookup to direct DNS resolution of a subdomain to the cluster. If you enable GNS, then you must have a DHCP service on the public network that allows the cluster to dynamically allocate the virtual IP addresses as required by the cluster.

Validation is performed... and accepted!

As a result of enabling GNS, I can no longer assign VIP addresses to my cluster node names. These addresses are assigned by GNS from now on. Of course, when I wouldn't have enabled GNS, I were able to assign the addresses in this screen.

Regular SSH connectivity test

Network Interface Usage specification. Should be no surprise for anyone experienced in 10g Clusterware.

Now, I will come to the choice of my storage solution. Before I started the installation I added another shared disk, enabled ASMlib on both servers and created an ASM Disk. Therefore, I can now choose ASM for storage.

Choose the diskgroup name and assign disks to the diskgroup.

Passwords. Be careful here! Passwords should comply with Oracle Recommendations, however you can ignore them. You will be notified when something is wrong or should be reviewed according to Oracle:

I am not using IPMI (Intelligent Platform Management Interface) so, I didn't select this option.

IPMI provides a set of common interfaces to computer hardware and firmware that system administrators can use to monitor system health and manage the system. With Oracle 11g release 2, Oracle Clusterware can integrate IPMI to provide failure isolation support and to ensure cluster integrity.
You can configure node-termination with IPMI during installation by selecting a node-termination protocol, such as IPMI. You can also configure IPMI after installation with crsctl commands.

Here you can assign different groups to specific management options.

ASM Database Administrator (OSDBA)

ASM Instance Administration Operator (OSOPER)

ASM Instance Administrator (OSASM)

Oracle has separated the privileges for each of these groups of administrators. Therefore, when I chose dba as group for each of these, I got a warning. Under normal (production operational) circumstances, I should have created three different groups and assigned these groups to different users on my system. Well, I have got only one user on my test environment: oracle, so what is the use of setting up three separate groups? Anyway, the idea is clear and it is good practice to follow these guidelines.

Specify the locations of the Oracle Base and Oracle Grid Home. Oracle Grid Home should be installed parallel to the Oracle Base, so not inside the Oracle Base!

File locations are being checked on both servers.

New feature: Maybe it is better to say there is an enhancement here. The system is being verified thouroughly for prerequisites. The result can be seen in the next screen shots:

Uhh. I have got some work to do here...
Nice new feature : the fixup script. Everything that Oracle can change in a script, a script is provided for by the installer. Just run the script and those checks that failed and are Fixable are being fixed. Don't worry about prerequisite checks before running the installer. Just let the installer do the work for you. It'll report everything you need to fix before installing. Just push the "Check Again" button as many times as you wish until everything is reported as passed.

Execute the runfixup.sh script:

...And Check Again...

Well, I didn't have enough memory and swap. Also, NTP wasn't started with the -x option... Who cares? - We'll see at the end of this weblog...

...summary...

Start the installation by clickin on Finish.

This is what you'll see during installation:

Tap, tap... :-)

...tap...!

We're almost there! Just run these two scripts and we're finished with the installation of Grid Infrastructure.

Well, that is what I thought. It took my virtual machines almost all night to finish the last script (root.sh in Grid Home). In fact, the script timed out after an hour or so, and the Virtual Machine practically froze... After having waited a couple of hours it even rebooted my entire machine (Including the Host OS!).

I will have to assign some more memory to the guests and see if I can finish the installation, or whether that is needed at all. However, it is weekend now, my wife and kids will not like me sitting behind my laptop all weekend. We'll be back next week...

11g Release 2 New Features for Grid Infrastructure

2009-09-04T09:05:00.003+02:00

Being a RAC fanatic, the new Oracle 11g Release 2 Grid Infrastructure is one of my main focuses to investigate. Here is what I found to be the new features of Grid Infrastructure, which by the way comprises of Oracle Clusterware and Oracle Automatic Storage Management (ASM). All of them are truly fantastic features that are screaming to be investigated... I'll do my best to cover some of them here in the near future.

Oracle grid Infrastructure
With Oracle grid infrastructure 11g release 2 (11.2), Automatic Storage Management (ASM) and Oracle Clusterware are installed into a single home directory, which is referred to as the Grid Infrastructure home.

Oracle Automatic Storage Manager and Oracle files
With this release, Oracle Cluster Registry (OCR) and voting disks can be placed on Automatic Storage Management (ASM).
This feature enables ASM to provide a unified storage solution, storing all the data for the clusterware and the database, without the need for third-party volume managers or cluster filesystems.
For new installations, OCR and voting disk files can be placed either on ASM, or on a cluster file system or NFS system. Installing Oracle Clusterware files on raw or block devices is no longer supported, unless an existing system is being upgraded.

Automatic Storage Management Cluster File System (ACFS)
Oracle Automatic Storage Management Cluster File System (Oracle ACFS) is a new multi-platform, scalable file system and storage management design that extends Oracle Automatic Storage Management (ASM) technology to support all application data. Oracle ACFS provides dynamic file system resizing, and improved performance using the distribution, balancing and striping technology across all available disks, and provides storage reliability through ASM's mirroring and parity protection.

ASM Job Role Separation Option with SYSASM
The SYSASM privilege that was introduced in Oracle ASM 11g release 1 (11.1) is now fully separated from the SYSDBA privilege. If you choose to use this optional feature, and designate different operating system groups as the OSASM and the OSDBA groups, then the SYSASM administrative privilege is available only to members of the OSASM group. The SYSASM privilege also can be granted using password authentication on the Oracle ASM instance.
You can designate OPERATOR privileges (a subset of the SYSASM privileges, including starting and stopping ASM) to members of the OSOPER for ASM group.
Providing system privileges for the storage tier using the SYSASM privilege instead of the SYSDBA privilege provides a clearer division of responsibility between ASM administration and database administration, and helps to prevent different databases using the same storage from accidentally overwriting each other's files.

Cluster Time Synchronization Service
Cluster node times should be synchronized. With this release, Oracle Clusterware provides Cluster Time Synchronization Service (CTSS), which ensures that there is a synchronization service in the cluster. If Network Time Protocol (NTP) is not found during cluster configuration, then CTSS is configured to ensure time synchronization.

Enterprise Manager Database Control Provisioning
Enterprise Manager Database Control 11g provides the capability to automatically provision Oracle grid infrastructure and Oracle RAC installations on new nodes, and then extend the existing Oracle grid infrastructure and Oracle RAC database to these provisioned nodes. This provisioning procedure requires a successful Oracle RAC installation before you can use this feature.
See Also: Oracle Real Application Clusters Administration and Deployment Guide for information about this feature.

Fixup Scripts and Grid Infrastructure Checks
With Oracle Clusterware 11g release 2 (11.2), the installer (OUI) detects when minimum requirements for installation are not completed, and creates shell script programs, called fixup scripts, to resolve many incomplete system configuration requirements. If OUI detects an incomplete task that is marked "fixable", then you can easily fix the issue by generating the fixup script by clicking the Fix & Check Again button.
The fixup script is generated during installation. You are prompted to run the script as root in a separate terminal session. When you run the script, it raises kernel values to required minimums, if necessary, and completes other operating system configuration tasks.
You also can have Cluster Verification Utility (CVU) generate fixup scripts before installation.

Grid Plug and Play
In the past, adding or removing servers in a cluster required extensive manual preparation. With this release, you can continue to configure server nodes manually, or use Grid Plug and Play to configure them dynamically as nodes are added or removed from the cluster.
Grid Plug and Play reduces the costs of installing, configuring, and managing server nodes by starting a grid naming service within the cluster to allow each node to perform the following tasks dynamically: Negotiating appropriate network identities for itself
Acquiring additional information it needs to operate from a configuration profile
Configuring or reconfiguring itself using profile data, making hostnames and addresses resolvable on the network Because servers perform these tasks dynamically, the number of steps required to add or nodes is minimized.

Intelligent Platform Management Interface (IPMI) Integration
Intelligent Platform Management Interface (IPMI) is an industry standard management protocol that is included with many servers today. IPMI operates independently of the operating system, and can operate even if the system is not powered on. Servers with IPMI contain a baseboard management controller (BMC) which is used to communicate to the server.
If IPMI is configured, then Oracle Cluster uses IPMI when node fencing is required and the server is not responding.

Oracle Clusterware Out-of-place Upgrade
With this release, you can install a new version of Oracle Clusterware into a separate home from an existing Oracle Clusterware installation. This feature reduces the downtime required to upgrade a node in the cluster. When performing an out-of-place upgrade, the old and new version of the software are present on the nodes at the same time, each in a different home location, but only one version of the software is active.

Oracle Clusterware Administration with Oracle Enterprise Manager
With this release, you can use Enterprise Manager Cluster Home page to perform full administrative and monitoring support for both standalone database and Oracle RAC environments, using High Availability Application and Oracle Cluster Resource Management.
When Oracle Enterprise Manager is installed with Oracle Clusterware, it can provide a set of users that have the Oracle Clusterware Administrator role in Enterprise Manager, and provide full administrative and monitoring support for High Availability application and Oracle Clusterware resource management. After you have completed installation and have Enterprise Manager deployed, you can provision additional nodes added to the cluster using Enterprise Manager.

SCAN for Simplified Client Access
With this release, the single client access name (SCAN) is the hostname to provide for all clients connecting to the cluster. The SCAN is a domain name registered to at least one and up to three IP addresses, either in the domain name service (DNS) or the Grid Naming Service (GNS). The SCAN eliminates the need to change clients when nodes are added to or removed from the cluster. Clients using the SCAN can also access the cluster using EZCONNECT.
SRVCTL Command Enhancements for Patching
With this release, you can use srvctl to shut down all Oracle software running within an Oracle home, in preparation for patching. Oracle grid infrastructure patching is automated across all nodes, and patches can be applied in a multi-node, multi-patch fashion.

Typical Installation Option
To streamline cluster installations, especially for those customers who are new to clustering, Oracle introduces the Typical Installation path. Typical installation defaults as many options as possible to those recommended as best practices.

Voting Disk Backup Procedure Change
In prior releases, backing up the voting disks using a dd command was a required postinstallation task. With Oracle Clusterware release 11.2 and later, backing up and restoring a voting disk using the dd command is not supported.
Backing up voting disks manually is no longer required, as voting disks are backed up automatically in the OCR as part of any configuration change and voting disk data is automatically restored to any added voting disks.

Oracle Database 11g Release 2 available

2009-09-01T20:54:00.002+02:00

I just found out that Oracle Database 11g Release 2 for Linux is available for download.
I am downloading it now, hopefully I will find some time to investigate some of those new features. I have heard there are a bunch in the RAC and ASM area. I can't wait to get my fingers on it...

Be back soon!

OGh DBA Day in Utrecht, The Netherlands

2009-08-27T08:36:00.005+02:00

On November 3, 2009, the OGh (Oracle Gebruikersclub Holland - An independent Oracle user group in The Netherlands) will organize a DBA day. Topic of the day is High Availability.

I will be doing a presentation on the subject of recovering a corrupted database without restoring the backup. A Demo will be included, as well as pros and cons, opportunities and limitations.

The rest of the day will be defined soon, so for the moment no further information is available.
Check back soon for more details!

New Feature Added: E-Mail Notification

2009-08-20T10:19:00.002+02:00

I added a new feature to my blog today: E-Mail Notification.
If you are interested in getting updates from this Weblog, simply subscribe using the widget at the side-bar of this page. You will get an e-mail as soon as new posts are published.
Happy reading and have a wonderful day.

Certified, at last!

2009-07-03T11:31:00.003+02:00

Last year I took the beta exam for E-Business Suite Release 12, Install, Patch and Maintain Applications. Unfortunately I failed that exam, but I didn't prepare at all. In fact, I found out about the beta exam one day before the beta program ended...
Now I took some more time to prepare for the exam and passed with 82% score.
So, as of today I am officially allowed to call myself: Oracle E-Business Suite Release 12 Database Administrator Certified Professional.
The exam was very appropriate: Proper level of difficulty, good questions, some pretty hard, I must say.
If you want to prepare: Take the course (Install Patch and Maintain Applications), learn all the new features of R12, the new file locations (logfiles, etc), and make yourself familiar with the managment tools around R12. Then you should be able to pass without too much problems.

E-Business Suite R12.1.1 - Which database version?

2009-06-25T15:57:00.007+02:00

Fooling around with E-Business Suite R12.1.1 installation this week led to some confusion about which database version comes with the distribution.

The first surprise came when we started Rapid Install and unfolded the components to be installed, as outlined in the welcome screen:

Huh? 9.2.0.5??? I thought 11.1.0.7 was to be installed with R12? 9.2.0.5 isn't even certified.

After we entered all the information, the installation was kicked off, and we saw the following information on our screen:

Still, not quite the version that we were expecting, but coming close, at least.
After the installation was finished, we were - again - mislead, as you can see in this and the following screenshot:

Luckily, checking the database version from within E-Business Suite (Help - About), finally told us the truth, and nothing but the truth:

So it is true! Oracle E-Business Suite Release 12.1.1 comes with 11.1.0.7 database!

Upgrading 11.5.9 base to R12.1.1: Do we need CU2 or not?

2009-06-11T08:38:00.004+02:00

We have recently started the process of upgrading one of our customers environments to R12.1.1. The current version of the E-Business Suite is 11.5.9 base.
First thing we did was to investigate the Oracle Applications Upgrade Guide: Release 11i to Release 12.1.1.
This document states (Chapter 1, Supported Upgrade Paths, on page 1-1) that it is possible to upgrade from 11.5.9 base, directly to R12.1.1.
However, the text on page 2-29 (Prepare for the Upgrade, Step 4: Migrate or upgrade your database to at least Oracle 10g Release 2) can be a bit misleading. The text states the following:

Oracle Applications Release 11.5.9 (with CU2 applied) and Release 11.5.10 (with CU2applied) are certified for use with Oracle 10g Release 2, while 11.5.10 (with CU2) iscertified for use with Oracle 11g Release 1. If you have not already done so, you canupgrade your production database to at least 10g now, before the upgrade downtime.Follow the instructions in Database Preparation Guidelines for an Oracle E-Business SuiteRelease 12.1.1 Upgrade (Doc ID: 761570.1).

The misleading part (in a way) is the first line. The conclusion one can draw here is that is is necessary to first apply CU2 to 11.5.9 base before upgrading the database to 10gR2. We almost did, anyway. Further investigation in the provided note (761570.1) showed us that we need to apply the Interoperability Notes for Oracle 10gR2 database with 11i Applications (Note 362203.1), but skip the following sections:

Verify software versions
run adgrants.sql
implement and run Autoconfig
re-create grants and synonyms
apply Oracle Receivables patch
restart Applications Server processes

The first mentioned section (Verify software versions) is the section that holds the instruction to apply CU2 to 11.5.9.

In short, if

your E-Business Suite is at level 11.5.9 base or CU1 and
your E-Business Suite database is 9.2.0.x,

in the process of upgrading to R12.1.1, there is no need to apply CU2 to 11.5.9 before upgrading your database to 10gR2.

PS. It is not possible to upgrade your 11.5.9/9.2.0.x database to 11gR1. You will not use the 11gR1 ORACLE_HOME, which is delivered with your R12.1.1 techstack installation during the upgrade. If you want to upgrade your database to 11gR1, you can do so after the upgrade to R12.1.1 has completed.

Password Policies in Oracle E-Business Suite

2009-05-28T13:34:00.006+02:00

One of my customers is challenging the possibility to enforce strong passwords in E-Business Suite (Release 12). Using the generic User Define Form you can define when a password expires, but that is more or less all you can do from that screen. In order to enforce an advanced Password Policy, you should go to the Profile Options:

Signon Password Failure Limit
The Signon Password Failure Limit profile option defines the maximum number of login attempts before the user’s account is disabled.

Signon Password Hard to Guess
Set this Profile Option to Yes to ensure that they will be "hard to guess."
A password is considered hard-to-guess if it meets this requirements:
• The password contains at least one letter and at least one number.
• The password does not contain the username.
• The password does not contain repeating characters.

Signon Password Length
Signon Password Length defines the minimum length of the password. Te default is 5 characters

Signon Password No Reuse
This profile option specifies the number of days before any previously given password can be reused.

Signon Password Case
Set this profile option to 'Sensitive' to make the password case sensitive (it defaults to 'Insensitive in 11i, apparently, it defaults to 'Sensitive' in R12.1.1).

In this example, Users will have to enter a case sensitive password, they are not allowed to enter more than 3 wrong passords, the password must be hard to guess (see above), the lenght is set to at least 8 characters and cannot be used again for at least a year after it has expired.

In the Define User screen we can set the Password Expiration to either
• Days (see example),
• Accesses (the number of logins) or
• None.

Combining the profile options with the Password Expiration will give you a robust password policy for Oracle E-Business Suite.

R12.1.1 - New Technology Features at a Glance

2009-05-26T21:45:00.006+02:00

Here is a quick list of the new technology features for E-Business Suite Release 12.1.1:

Versions

Database
Version 11.1.0.7 comes pre-packaged with E-Business Suite Release 12.1.1.
Version 10gR2 is still supported, for upgrades

Application Server
10.1.2.3 for Forms And Reports
10.1.3.4 for Java code and OAF, HTTP Server and OJSP

Third Party Technology
Java JDK version 6.0
Native Java Plugin for Client side browser) version 6.0 (5.0 still supported for upgrading customers).

Technology Components in APPL_TOP
JDeveloper runtime libraries version 10.1.3.4
Oracle BI Beans version 3.1.1.7
Oracle Thin JDBC Drivers version 11.1.0.7

Externally Installed Technology Certified with 12.1.1
Oracle AS 10g Portal 10.1.4.2
Oracle AS 10g Single Sign-on 10.1.4.3
Oracle AS 10g Discoverer 10.1.2.3
Oracle Collaboration Suite 10.1.2
Oracle Enterprise Manager 10.2.0.2

* The above require separate installation, the products mentioned don't come packaged with the distro.

Technology Configuration Management

Autoconfig has been improved greatly in this new release of Oracle E-Business Suite:

Profiler Mode
This mode has been added to the 12.1.1 AutoConfig. When you run AutoConfig in this mode, it can generate a performance report containing timing details about each script that is run by AutoConfig.

Parallelization
Autoconfig can be run in parallel on multiple nodes, reducing downtime.

Control Dependency Management
Redesigned service and service group definitions in the context file, enabling a service to be in different service groups and easing the addition of new services. Additional support for dependencies between service group. Introducing the possibility to enable and disable specific OC4J instances on the Application Tier Servers.

adchkcfg
Enhanced to report on file system and database changes prior to running autoconfig. The report has a R12 L&F.

adbldxml on the Database Tier
This utility can now be used to create new context files on the database tier to facilitate database upgrades and cross-platform migration. This feature was available in 11i, but removed in R12.0, but luckily reintroduced again in R12.1.1

AutoConfig Search Utility
Run from the command line, this utility can be used to get detailed information on context variables and the templates where these are used. Nice detail: If you don't know the entire variable name, it also accepts part of the variable name.

Technology Stack Inventory Validation Report
This utility validates the TechStack Inventory, similar to the TechStack Validation utility. The resulting report shows component versions, installed patches and patch sets.

Application Tier File System sharing
There is now support for sharing the Application Tier File System amongst multiple Oracle E-Business Suite instances. In Release 12.0 the APPL_TOP could be shared, because the Instance Home was introduced. Now there is support to share the entire Application Tier File System (including the Application Server Tech Stacks). Refer to Metalink Note 384248.1.

Enhanced Support for DMZ deployments
New demilitarized zone (DMZ) deployment options added, like support for forward proxies, reverse proxies without external web tiers, and the option to use hardware load-balancers without an external web tier.
More information in Metalink Note 380490.1

Application Tier Load Balancing
Enhancements in support for major load balancing methods: DNS, HTTP Layer and Native OC4J. Refer to Metalink Note 380489.1

Network Traffic Encryption
There is now Autoconfig support for securing the main communication with SSL: Desktop To WebServer (HTTPS), WebServer to JVM (AJPS), JVM (and other technology processes) to database (Advanced Security or Encrypted SQL*Net). See Metalink Note 376700.1

Oracle Connection Manager
AutoConfig now supports Oracle Connection Manager with R12.1.1. Oracle Connection Manager is a security tool acting as a proxy server that forwards connection requests to database servers. For more information, see Metalink Note 558959.1.

One option that I cannot leave unnoticed (taken from Steven Chan's weblog):
The R12.1.1 Rapid Install allows you to upgrade to 12.1.1 from EBS 11.5.9, 11.5.10, 11.5.10.CU1, and 11.5.10.CU2. That is one cool feature to explore. Will do that soon, hopefully.

Come back soon!

E-Business Suite Release 12.1.1 - It is working!

2009-05-26T21:28:00.003+02:00

Yesterday morning I started my installation of Oracle E-Business Suite Release 12.1.1. I was very impressed by the installation, even though I performed the installation on a VMware guest Oracle Enterprise Linux 5.3, running with only 1.5GB of memory, from my external USB disk. Installation started around 8:15am and finished around 12:45pm: 4.5 hours. Not bad for an installation on an external USB drive...

I simultaneously discovered the solution to an issue with the R12 Webserver in combination with OCFS2 I reported upon about two years ago in one of my articles on the AMIS Technology Blog. I didn't realize it until I found out that, against my expectations, after the installation, surprisingly the webtier started without problems. I thought of two possible causes for this:

1. OCFS2 has improved, or
2. The new version of the Oracle Internet Application Server (10.1.3.4) has improved.

Neither of the two turned out to be the case:

During the Rapid Install I (to be honest, accidentally) configured the Instance Home ($INST_TOP) on one of my local file systems. The Instance Home contains the E-Business Suite Instance specific configuration files, log files and other files that are specific to its particular instance. Therefore, this Instance Home doesn't require to be shared. There is nothing wrong putting it on a local file system.
Now, when defining the Instance Home to a local file system, also brings the location where the Apache Web Server wants to create this particular file ($INST_TOP/logs/ora/10.1.3/Apache/mm.XXXX) to the local file system. No need for symbolic links or whatsoever.

In my next article I will be discussing some of the new technological features of R12.1.1.

E-Business Suite R12.1.1 installation impression

2009-05-26T10:04:00.003+02:00

Two weeks ago I was at Collaborate09 where Charles Phillips announced the release of E-Business Suite 12.1.1. Exciting news, since 12.1.1 comes with a number of enhancements screaming for attention.

First of all, the most eye-catching enhancement is the pre-packaged 11gR1 (11.1.0.7) database. Second is the inclusion of all the Critical Patch Updates.
There is also a number of updates on the Application Servers for Forms (version is now 10.1.3.4)

As soon as I came back home, I downloaded the 12.1.1 software and created a VM to test the installation. I installed Oracle Enterprise Linux 5.3 with a bunch of storage... You definitely need some storage for 12.1.1! Vision Demo Database only is over 200GB!
Starting with the Oracle Applications Installation Guide: Using Rapid Install, you will be guided through the installation process. Since I am on OEL5.3 32-bit, I was pointed to Metalink Note 761564.1 - Oracle Applications Installation and Upgrade Notes Release 12 (12.1.1) for Linux x86.
In this document you can find all the details about installing EBS 12.1.1 on Linux x86:

RPMS required to install 12.1.1:

openmotif21-2.1.30-11.EL5.i3861
xorg-x11-libs-compat-6.8.2-1.EL.33.0.1.i386
binutils-2.17.50.0.6-6.0.1.i3862

The above RPMs can be downloade from oss.oracle.com.
The list of RPMs down here can all be found on your distribution media of OEL5.3:

compat-glibc-2.3.4-2.26
gcc-4.1.2-14.el5
gcc-c++-4.1.2-14.el5
glibc-2.5-123
glibc-common-2.5-123
glibc-devel-2.5-12
libgcc-4.1.2-14.el53
libstdc++-devel-4.1.2-14.el5
libstdc++-4.1.2-14.el53
make-3.81-1.13
gdbm-1.8.0-26.2.13
libXp-1.0.0-8.1.el5
libaio-0.3.106-3.23
libgomp-4.1.2-14.el5
sysstat-7.0.0-3.el5
compat-libstdc++-296-2.96-138
compat-libstdc++-33-3.2.3-61

If you are performing a fresh install of 12.1.1, like I did, you will also need some RPMs for 11g database:

elfutils-libelf-devel-0.125
elfutils-libelf-devel-static-0.125
libaio-devel-0.3.106
unixODBC-2.2.11
unixODBC-devel-2.2.11
kernel-headers-2.6

Now all required RPMs have been installed, take a look at the kernel settings:

Add or adjust the following parameters in /etc/sysctl.conf:

kernel.sem = 256 32000 100 142
kernel.shmall = 2097152
kernel.shmmax = 2147483648(*)
kernel.shmmni = 4096
kernel.msgmax = 8192
kernel.msgmnb = 65535
kernel.msgmni = 2878
fs.file-max = 131072
net.ipv4.ip_local_port_range = 10000 65000(**)
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 262144

(*) least value. Half the memory if that is more
(**) I had 1024 65000 and it worked too

Apply these settings by saving the /etc/sysctl.conf and typing the command:
sysctl -p

Next, adjust the DNS resolver parameters in /etc/resolv.conf:
options attempts:5
options timeout:15

Next, make sure your hosts fully qualified domain name is mentioned as the first alias to the hosts IP address in /etc/hosts:
192.168.1.10 hostname.domain.name.com hostname

Make sure that /etc/sysconfig/network contains the fully qualified domain name as your hostname.

In /etc/security/limits.conf you should set some limits for the user you are installing EBS under:
oracle hard nofile 65535
oracle soft nofile 4096
oracle hard nproc 16384
oracle soft nproc 2047

Next, apply patch 6078836 for Oracle or RedHat Enterprise Linux 5. This is an important prerequisite patch in order to prevent compilation errors during installation.
After you have applied this patch, perform the following as root:
# unlink /usr/lib/libXtst.so.6
# ln -s /usr/X11R6/lib/libXtst.so.6.1 /usr/lib/libXtst.so.6

Now we're ready to install.
Installation is currently underway.
I'll be back soon to update on my findings.

Collaborate 2009: Mixing E-Business Suite and Pleasure

2009-04-26T08:57:00.003+02:00

So, This is the last week before I will be leaving for Orlando, FL. Next saturday (may 2) I will be taking off to join Collaborate09. I am excited and a little nervous too. It will be the first time to present at a large scale conference like this.
If you are interested: Look out for session #1037, Mixing E-Business and Pleasure on Tuesday, May 5, 3:15PM - 4:15PM in room W105A.
In this presentation I will share my experience with a large-scale E-Business Suite implementation with SOA integration (mainly BPEL). The business case will be discussed, what the customer wanted to achieve, and what issues and challenges the actual implementation brought.

I am really looking forward meeting you there.

Outsourcing your E-Business Suite environment - Part V

2009-03-10T13:58:00.004+01:00

In my previous articles in this series I explained what the key areas of focus would have to be when you consider outsourcing your Oracle E-Business Suite environment(s). This last article in this series covers some of the questions you could or should ask to your potential partners.

Now that you have a rough idea about the issues to bear in mind when outsourcing your E-Business Suite environment(s), it is time to look for a partner that you can do business with. In order to make a proper judgement, it is essential to know which details are important to your business. Ask questions on those details. They are important to you, so you must find the answers with your partner, and weigh them on the scale of importance.

Criteria

Here you can find a list of criteria you might find useful:

Every customer should be logically separated from other customers
Possibility to filter traffic between systems of the same customer
In case of an attack/intrusion it should be possible to quickly block traffic
Detection by means of a network intrusion detection system should lead to action in less than 3 minutes
Possibility to guarantee bandwidth
Possibility to regulate bandwidth
Possibility to deliver connections, by fixed lines as well as VPN via Internet
Data Center has more than 1 public exchange utilizing multiple carriers
Multiple access paths to the public exchange are possible, physically as well as logically, so in case of interruption of service altenate paths could provide required service
Termination is separated from the server room, so physical access is not required for assembly
Separated LANs for different purposes (backup/recovery, management, public, cluster heartbeat)
Failover datacenter at least 5 miles away
Support for QoS (Quality of Service)
Trunking Technology should be supported
Local LAN is redundant, including switches, SAN switches, cables, power supplies
Redundant Airconditioning with sufficient capacity
ISO27001 compliancy/certification
Availability of No-Break
External Power Generators are available, are tested monthly and meet capacity requirements
Growth of XXX% can be facilitated
Hardware defects can be solved within SLA times
Near Real-Time information about systems, load and amount of traffic available online or can be made available online, included in regular pricing
Monitoring is taking place for intrusion, fire, power failure, high/low voltage, external power generator failure, UPS/No-Break failure, temperature, humidity, breaker trips and leakage
24x7 security available on the premises
24x7 access for authorized personell of customer
Backup facilities available and optionally availability to take backups to multiple tape units
E-mail facilities are available for outbound mail from specified servers
Physical access to hardware is separated from possible other customers to prevent other customers having access to our hardware
Capacity planning should be in place, for at least 12 months
Operations Staff should be adequately trained
Credentials should not be stored human readable, i.e. no hard coded credentials in scripts etc.
Cleaning and Archiving procedures in place for log files etc.
All access to systems must be logged and should be tracable to a personal account
Availability of centralized authentication system with LDAP or NIS
Servers should be addressed through a servername using Fully Qualified Domain Names
OS Storage needs to be mirrored
Remote Console Access should be possible
A representation of current configuration and hardware as well as Operating System should be delivered when asked (CMDB report)
Following Operating Systems and distributions should be supported: (fill in your required Operating Systems)
All machines are configured with swap space according to best practices
All File Systems are configured with either RAID5, RAID1 or RAID1+0
All provided slices of storage can be resized dynamically
Availability of a supported platform for backups
Every Unix machine is provided with ping, traceroute, lsof, top, truss, strace, sar or applicable equivalences for the OS in place
Telnet services are disabled, ssh access should be possible
A Network Time Protocol Server is available for every system and is redundant
An authorization and maintenance process is in place for all privileges
Security audits are taking place on a regular basis, conform ISO27001
Gigabit Ethernet is standard
Relevant information on all layers is stored onto which trend-analysis can be performed
On at least a monthly basis, investigation is done regarding patches to be applied and proposed as change to customer
Weekly SLA Meetings
Weekly Service Level Reports
NIS and FTP services are disabled
UIDs and GIDs of non-system OS users and groups should always be identical across systems
Possibility to perform routine maintenance by means of sudo
Availability and Performance Management should be in place
Architecture needs to be provided with a flexible storage solution for allocation, deallocation and copying of data and efficient backups
Backup data should be kept online on a remote location, at least 5 miles away from the primary data center
It should be possible to put backups on tape, to deliver at customer site
Internet connected machines should be placed in a DMZ
The meeting structure being used, by means of communication and consultation on strategical, tactical and operational level is formatted by a governance model
Support needs to be provided in the … language
A situation is designated as a crisis at initiative of customer; in such a situation supervision will be accepted from the by customer assigned crisis manager
At least 1FTE with knowledge of the environment is available at all times to support and investigate opon request
Customer can take backups to disk and optionally to tape at a later time, facilitated by hosting provider
It should be possible to house or host optional “external” hardware, like customer owned systems

Key Performance Indicators

Next, you should have a set of Key Performance Indicators (KPIs), that will have to be mapped to the criteria. You should have at least the following areas:

General
Availability
Incident Management
Reaction Time, Resolution Time, Down Time, etc.
Change Management
Maintenance Windows, Change Qualification Time, etc.
Configuration Management
CMDB, Monitoring, etc.
Operations Management
Backup, Maintenance, etc.
Service Management
Reports, meetings, etc.

Questions

Based on the above, you could ask your party

what their vision is on separation of responsibilities and how they would realize this vision,
which risks they identify and how they would minimize these,
their ability to meet the criteria as stated,
their ability to meet service levels mentioned in the key performance indicators outlined by you,
to share a transition plan and approach for application management takeover and / or migration of current to new infrastructure,
to identify cost drivers for accounting services and products, based on the separation of responsibilities and conform the given criteria,
identify the monthly cost per cost driver,
how they detect intrusions,
how they guarantee bandwidth,
how they guarantee connection availability,
what time they need to establish a system,
what possibilities there are to customize environments in terms of:
kernel parameters,
operating system limits,
startup scripts,
disks and partitions,
mountpoints
which security certifications they have,
whether they have multiple datacenters,
what their vision is on capacity, availability and performance management and how that translates to practice,
if they can provide an example of an incident report, if possible for multiple priorities/severities,
if they can provide an example of a change report,
what their definition is of a problem,
whether they have experience managing/hosting
Oracle Database environments and for which versions,
Oracle E-Business Suite environments and for which versions,
Oracle Application Server environments and for which versions,
If they can provide any references for any of the above with information about:
Version Size
Number of users
Number of transactions
Availability.

This list is not complete, but it can give you a start to identify the kind of questions you might want to be answered, in order to make a proper judgement.

I hope this series has given you some clearer view on the topic of outsourcing your environment. If you have any questions left, you can always drop me a comment.

Outsourcing your Oracle E-Business Suite environment - Part IV

2009-02-26T15:15:00.005+01:00

In the previous articles in this series about outsourcing E-Business Suite environments I discussed the question why, the more technical aspects and the supporting services. This article will cover the topics availability, disaster recovery and security.

Availability

Load Balancing - Load Balancing techniques are used for two reasons: To increase availability and to increase capacity.

Application Server
On the application server level of E-Business Suite, one can choose to implement two kinds of load balancing: Forms based (11i) and HTTP based (11i and r12). In order to run forms based load balancing, one server needs to be designated as Forms Metrics Server, the remaining servers will be Forms Metrics Clients. On each of the servers the Forms Server needs to be running. All clients will be connected to the Forms Metrics Server, which diverts the connection to either of the forms servers in the application tier. This is a basic form of load balancing, because it is based on a round-robin principle. The alternative is HTTP Load Balancing, which requires Forms Servlet Mode to be in use. In this configuration, the forms server is running through a servlet, which runs under the http server. Therefore, there is no need to use other ports than the port the http server is using. A side effect is increased security. Another requirement is to use a separate load balancer. This load balancer will be configured with a central IP address (resolvable by a DNS name). The connections will be spread over the available application servers. This can be done on a round-robin principle, but can also be based on actual system load, like with Cisco ACE technology. One thing to keep in mind, which is of great importance when implementing load balancing is to ensure that Session Stickiness/Persistence (IP or cookie based) is implemented.
It is also possible to implement DNS based load balancing. In this configuration, the DNS server has got multiple IP addresses for the same hostname and will randomly reply a possible IP address to any DNS request for an application server. This requires some additional configuration on the Application Tier, like JServ load balancing and defining OProcMgr nodes (web nodes) in your environment (using the context editor).
Refer to metalink note 217368.1 for implementing load balancing for E-Business Suite

Database Server
Increasing availability on the database server can be established using Real Applications Cluster technology. Implementing Real Applications Cluster will require Oracle Clusterware, and takes some of the system resources for cluster node intercommunication. However, the increase in availability will more than compensate for this. I do need to say that no matter how much I appreciate RAC, it adds to the complexity of your infrastructure and you will definitely need resources in your team or at your hosting provider (depending on who is going to manage/maintain the E-Business Suite) capable of managing and maintaining Real Applications Cluster environments. Otherwise you may end up with a system with lower uptime compared to a single instance database environment.

Disaster Recovery
Dataguard – This topic adds up to the previous one, availability. If your hosting provider can provide multiple data centers, it might be worthwhile investigating the possibilities of implementing DataGuard. With E-Business Suite you can establish a physical standby database on a remote location that can be switched over to when the primary database fails for whatever reasons. For more information on implementing DataGuard ee Metalink Notes 216212.1 and 403347.1 for Release 11i and Metalink Note 452056.1 for Release 12.

Data Replication – If Dataguard is not feasible, you should at least be given the possibilities for data replication. Make sure your hosting partner is able to replicate your business-critical data to a remote data center, so you, or your hosting partner can rebuild the entire environment on another location in case of a site failure.

For any DR solution, you should be able to quantify the maximum downtime you find acceptable. In order to be able to do this, it may be required to estimate the cost of downtime per time-unit. Take into consideration that your company should have the resources to sustain the damage of this downtime. Make sure your hosting partner can live up to the required level of service. After all, it is your business-critical data.

Security
Talking about business-critical data, you don’t want anyone that is not supposed to be there strolling around in your environment. Therefore it is of greatest importance that your hosting partner can guarantee the highest level of security. The International Standards Organization (ISO) has a certification for this: ISO-27001:2005. Ask for this certification at your partner of choice. This ensures that your partner has been audited on a number of controls dealing with information security.
Various security issues need to be covered when you want to outsource your environment. This can be issues that you as a customer may require, but it can also be issues that are required by your hosting partner. It is essential to work these out before starting your contract, or you may be surprised your partner doesn’t provide a solution you want to be implemented because of their tight security, or you will become personably responsible for security measures your partner can or will not provide.

Hardening – your systems need to be secure enough by themselves. This means that e.g. you don’t want to allow direct root access from a remote location, sudo lists to limit the amount of users that may become root, sudo command lists, to limit the commands that can be run with root privileges, limiting the services to those that are necessary to run the environment.

External Access – Remote OS access should be regulated by an external authentication system like an Active Directory system. If this is not possible, you should require vLANs that separate your environment(s) from others. It can also be arranged with Access Control Lists, but they should be combined with an AD solution.Intrusion Detection – No matter how secure your environment is hosted, it will always be possible for someone to try and attack your environment. For this, it is important to have an Intrusion Detection system in place. This system alarms when it detects uncommon activities on the environment, indicating someone or some program is trying to attack the system.

E-Business Suite Security - Ask your outsourcing partner what they would do to secure your E-Business Suite. It is of great importance. You might have a secure Operating System, but your application that runs on it must be secure as well, because it might be located at a remote site, in a hosting center, along with various other applications from who knows where. Security may very well be the top priority on your list. Refer to Metalink Note 189367.1 - Best practices for securing Oracle E-Business Suite for directions.

I hope to publish a questionnaire to ask a potential outsourcing partner as a conclusion to this series.

Further Tuning of OC4J Containers in SOA Suite 10.1.3.3

2009-02-25T11:49:00.005+01:00

In one of my recent posts I discussed a performance problem with OC4J Containers being restarted in SOA Suite. This week I had to go back, the environment had become a lot more stable, however similar problems still occurred.
Conclusion: The system was more stable, however the cause of the problem had not been taken away yet, so further investigation was needed.

This time I did some more investigation in tuning java options (opmn.xml) and increased the heap space to 2GB, hoping this would work out, however to no result: The problems persisted.

Then, my eyes fell on the error_log of the http daemon. I wondered what the cause of all of those "oc4j_socket_recvfull timed out" messages could mean...

There is a number of Metalink Notes about this issue, however the solutions I checked were not solving our issues.

Then I realized that the environment was deployed on AIX 5.3. I have worked for IBM and with AIX for a couple of years. I have seen similar issues on other environments (like Oracle E-Business Suite and RAC) and I still had a note somewhere about network parameter tuning on AIX. I checked these on the server, and found and fixed the following parameters, which seemed to have solved the issues:

rfc1323 - This defaults to 0 on AIX, and should be set to 1. By default, the TCP window size is limited to 65536 bytes (64 K) but can be set higher if the rfc1323 value is set to 1 (see tcp_recvspace value).

tcp_sendspace - The TCP Send Buffer. This defaults to 16384 bytes and should set to a higher value for Oracle Application Server (OHS). I set it to 266140.

tcp_recvspace - The TCP Receive Buffer. This also defaults to 16384 and should be set to a higher value for OHS. I set it to 266140, similar to tcp_sendspace. In combination with rfc1323, this enables a connection to negotiate a larger TCP window. If you are sending data through adapters that have large MTU sizes (32 K or 64 K for example), TCP streaming performance might not be optimal unless this option is enabled because a single packet will consume the entire TCP window size. Therefore, TCP is unable to stream multiple packets as it will have to wait for a TCP acknowledgment and window update from the receiver for each packet.

tcp_nodelayack - Value should be set to 1 for Oracle applications. The tcp_nodelayack option prompts TCP to send an immediate acknowledgement, rather than the usual 200 ms delay. Sending an immediate acknowledgement might add a little more overhead, but in some cases, greatly improves performance. Performance problems have been seen when TCP delays sending an acknowledgement for 200 ms, because the sender is waiting on an acknowledgment from the receiver and the receiver is waiting on more data from the sender. This might result in low streaming throughput.

Outsourcing your E-Business Suite environment - Part III

2009-02-23T10:55:00.005+01:00

In this third article about outsourcing E-Business environments I will cover the supporting Services that you may need for your environment.
Read the first and second article in this series on Outsourcing your E-Business Suite

Supporting Services
Oracle E-Business Suite is a large application suite with many functionalities. These functionalities sometimes depend on external services that it needs in order to provide functionality to a user or an organization. The Oracle E-Business Suite has a three tier model, i.e. a database, an application and a client tier. The connectivity between the tiers is of essential importance. Therefore, a number of measures need to be taken in order to provide services to enable connectivity according to requirements one is setting for their E-Business Suite.

DNS
Domain Name System - DNS is of great importance if you want your hostname resolution taking place centrally. However, your hosting provider can give you two options: to use their general DNS Servers, or to establish a “local” or customer dedicated DNS service. The latter will mean that a server (preferably more) in your infrastructure needs to be designated as a DNS server, and this DNS server needs to be maintained. One of the key questions here is whether you want your application servers to be able to resolve addresses within the companies own network. Your companies network will be – in a way – extended to the hosting partner, by means of a VPN or WAN connection. The question is whether your application servers need to be able to resolve host names in your own network. If that is the case, you will probably need your own DNS servers. You can dedicate specific servers for this task, but usually a shared server will suffice.

SMTP
Simple Mail Transfer Protocol - Oracle E-Business Suite is often used with Oracle Workflow. One of the major advantages of Oracle Workflow is that tasks can be assigned to people, and they can automatically be notified that specific tasks are waiting to be completed. Usually, this notification is done through the Workflow Mailer. As the name says, this service uses e-mail functionality for notification. If you are using Workflow Mailer, you will require a SMTP server to be available to the application server, in order to be able to send e-mails.Again, it is possible to have dedicated servers for this, but the SMTP service can be established on a server that for example also provides DNS Service.

DHCP
Dynamic Host Configuration Protocol - Usually DHCP is not in use for an Oracle E-Business Suite. The reason for me to mention the DHCP service is because when you are discussing the options with your possible partner of choice, they will ask you whether you need it. In my opinion, the answer should be no. The database servers will not use DHCP, neither the Application servers. Any other server should have a fixed IP Address, otherwise it would be a workstation, in my humble opinion.

SSL
Secure Sockets Layer - Unless you have a dedicated VPN or WAN connection, you should demand a kind of secure connection for your E-Business Suite. This can be done through SSL Accelerators, or by setting up SSL in the E-Business Suite. When you have a VPN connection, the connection is secured already, as is the case with a WAN connection, because both of them are extensions on your current network. Still, having SSL implemented for your production E-Business Suite is never a bad idea. With hackers around almost every corner these days, you can never be sure enough that the information that is exchanged between your clients and application servers is secured and encrypted.

AD
Active Directory – This supporting service can be used in two ways. One way is to configure Active Directory Services to control who can log on to the operating systems, the other way is to control access to the database and applications. AD services can provide access control to many users in your environment, and can even be used in a centralized, i.e. shared service configuration. This means that your hosting provider could use AD to control access for all of its customers, including you. By configuring the users in AD, the provider can grant and deny access to certain servers or parts of its infrastructure. The question whether you as a customer want this, is a whole different issue, but I will come back to this in my next article.

NTP
Network Time Protocol – Especially when you have a multi-node environment or cluster technology like Real Application Clusters, it can be of great importance that all of your servers have exactly synchronized time. In order to provide this, one of the servers in your infrastructure, or again a shared server in the hosting center of your partner should be configured as a NTP server. All of the servers in your infrastructure should be synchronizing their system clocks to this server via a small piece of software. It is also possible to use Time Servers on the Internet, but then your servers need to have external internet access (see outbound web services)

Inbound Web Services
This service enables accessibility from the outside internet. In many cases, your E-Business Suite will be used from within your own company infrastructure. There is a number of cases where you would have parties from outside your company needing access to your environment, for example iProcurement, Oracle Time and Labor, iStore, etc. may require your E-Business Suite to be available from the Internet, if the parties accessing your E-Business Suite don’t have access to your own network. Usually, the server providing this service is placed in a so-called Demilitarized Zone (DMZ) about which later in this article.

Outbound Web Services
If your servers providing the E-Business Suite application need access to internet, you will need Outbound Web Service. For example, you have need to configure your application server to access Metalink services for updates, or to upload configuration data which is used for the Oracle Configuration Manager. In these case your servers need access to Internet.

DMZ
Demilitarized Zone – This is an area in the network that has a lower level of security, in order to facilitate access from the Internet. In a normal situation the servers of your infrastructure are only available to your own organization and not to the outside world. If your requirements are such that access from the Internet should be possible, you would need a zone that is accessible from the internet (usually via a SSL port: 443), which holds the servers and applications that provide the necessary services. Usually these servers are a bit more hardened than the other servers, which are located behind one or more additional firewalls, i.e. a safer zone.

Most of the above supporting services will be needed to provide an Oracle E-Business Suite environment, regardless whether you outsource or not. However, when you consider outsourcing, you should have a very clear view on these aspects, all in one. Everything needs to be clear before you start an outsourcing project. Adding a service to your environment should never be a problem with any outsourcing partner, but it may be defined as a change to the project/contract, which will end up in additional (read: more) costs. Covering these at the beginning of the contract will save you money at the end of the day.
I know the above list is not yet complete. Aspects like availability, load balancing, disaster recovery and, last but not least, security have not yet been covered, though they may be even more important to your environment. I will cover these in my next article.

OC4J Containers getting restarted in OAS

2009-02-23T08:41:00.010+01:00

In the last weeks I have been called by a number of customers having problems with their newly installed Oracle Application Server, or rather Oracle SOA Suite.
I found more than once an issue with the timeouts on response times of these OC4J Containers.
All of the situations I was faced with were SOA Suites that had just been taken into production. The behaviour differed a little from case to case, but nevertheless the solution to the problem was found in the same solution for each of them.
OPMN manages the OC4J containers in Application Server (10g, that is). Because of this, it tries to ping all of the containers in order to check whether they are still alive. If the ping doesn't get returned quickly enough, it restarts the OC4J container and writes an error into the opmn.log file located under $ORACLE_HOME/opmn/logs. This can lead to various problems: If OPMN restarts the HTTP Server, the website will temporarily be unavailable which can be rather disturbing. If other OC4J containers get restarted, it can lead to various other errors like HTTP-500 (Internal Server Error) or other problems.
The solution is to tune the OC4J container ping parameters in the opmn.xml file found under $ORACE_HOME/opmn/conf:

For the http server, look for the tag process-type id="HTTP_Server" module-id="OHS"

. Add the following after this tag:

<process-set id="HTTP_Server" restart-on-death="true" numprocs="1">
<start timeout="300" retry="3"/>
<stop timeout="300"/>
<restart timeout="300" retry="3"/>
<ping timeout="60" interval="600"/>
</process-set>

Next, look for the tag within the HTTP Server definition and add the following lines:

<category id="ping-parameters">
<data id="ping-url" value="/"/>
</category>
<category id="restart-parameters">
<data id="reverseping-timeout" value="345"/>
<data id="no-reverseping-failed-ping-limit" value="3"/>
<data id="reverseping-failed-ping-limit" value="6"/>
</category>

For all other OC4J containers in the file, insert the same lines you added between the tags for the HTTP Server.
This will increase the timeout for the response of the OC4J container, giving it a little more time.

Outsourcing your E-Business Suite environment - Part II

2009-02-16T10:33:00.002+01:00

In the previous article in this series I pointed out why E-Business Suite is a perfect application to outsource.
This article will cover more of some technical aspects to give attention to while selecting an outsourcing partner.

Hardware Requirements - Your partner of choice should be able to meet the requirements you have set for your environment. Given the assumption that you have a running environment in-house, you don't want to be faced with a platform migration during an outsourcing project. It might very well be possible that you do want to migrate to another platform, but in general, an outsourcing project should not be the primary reason for this. There should be other reasons to migrate to another platform, and it might very well be more sensible to perform the platform migration in house, before starting the outsourcing project. The reason I am taking this approach, is because platform-migrating and outsourcing are two too big changes to your environment to combine in one project. Besides, If your partner of choice is able to provide a similar platform, the direct need for migrating may become obsolete, because they have better resources.

CPU - Many hosting providers, nowadays, offer blade technology. Often, the choices are limited to either Windows or Linux based solutions. The consequence is that the amount of processors is limited, often to 2 (be it single, dual or quad core). For smaller to mid-size environments this usually is no problem, however, whith larger environments environments can become quite complex. It must be said that with the newer CPU technologies, more powerful servers are achieved and the larger server solutions (like SUN Enterprise Servers, larger IBM pSeries, HP SuperDomes) become less in demand.

Network Interfaces - Especially with blade technology, it is of great importance to be able to influence your hosting partner when designing the network interface usage. Take an Oracle E-Business Suite environment running Real Application Clusters database technology. You would probably require a

Management LAN
Server LAN
Interconnect LAN
Storage LAN (if you are using NAS)

Most blades have a maximum of 4 network interfaces. Many hosting providers will tell you their default configuration is to team two sets of two interfaces, netting you to two network interfaces available to your applications. If you are using as storage LAN (NAS), you want as much bandwidth as possible, especially for your database servers. The Interconnect LAN must be non-routed and dedicated to the Interconnect traffic between the RAC nodes. The only option you have left is to share the Management vLAN and Server vLAN with the Storage vLAN over one Interface. In my humble opinion something you don't want.
Therefore, it might be necessary to move away from teamed interfaces and have dedicated interfaces. The fact that you are running RAC means you will have redundancy on a solution level, and the necessity for redundancy on NIC level will become less.

OS Requirements
Linux - In my humble opinion, for small to mid-sized environments I would definitely choose Linux (especially Oracle Enterprise Linux), because it is Oracle's platform of choice and because Oracle can support both application and OS. With business critical environments like E-Business Suite you don't want to spend time going from OS support provider to application support provider and vice versa, just because they are fingerpointing one another. Having the support centralized is one of the key features Oracle can provide when choosing Oracle Enterprise Linux.
However, do make sure your hosting provider is offering you the right type of Linux. Make sure you check the certification matrix, when your partner is offering you a standard Operating System Solution. One of the important details today is that 64-bit OS'es are offered frequently, however, for example E-Business Suite 11i only runs on a 32-bit OS, or at least a 32-bit kernel. It is possible to use 64-bit OS for the database, but not for the application. Things are a little different with E-Business Suite R12. R12 is certified against almost any 64-bit OS. No need to worry there.

Unix Same story here as above with Linux. Make sure your (or the proposed) architecture is in line with certification. Too many different flavors of Unix are available and there is too much to write about in a single article like this one. Certification is the key.

Windows To be honest, I have never seen an E-Business Suite running on Windows in my life, so writing about it would not be honest. What I do know is that you cannot just take a Windows Server and install Oracle E-Business Suite on it. It requires a couple of additional software packages in order to run.

Storage
NAS Especially when considering Linux, NAS is the way to go, when you want to have multiple application servers. I am a huge fan of SAN, however, when it comes to the application server, there is a lack of support on shared/clustered filesystems for the application tier of Oracle E-Business Suite. In fact, the only file system (as far as I am aware) that is certified with Oracle E-Business Suite is NFS, which is used in NAS technology. So, for sharing your APPL_TOP, do not consider OCFS, GFS or any other shared filesystem other than NFS. Oracle will help you when you run into trouble, but when problems cannot be reproduced on regular file systems, you will be diverted to the manufacturer of the file system. In production, I would never want to run on an architecture that is not fully certified and supported by the application.

SAN Probably the fastest storage solution available still. If your hosting provider is able to deliver SAN, use it for the database tier. First of all, when running on blade technology, this will save you an Ethernet port, which you can then use for different purposes. Second, performance is generally better with SAN (at least, that is according to my experience).

Backup / Recovery
NearLine Storage Your partner of choice should be able to provide remote storage. One of the key features of hosting your environment is to outsource the responsibility for data recovery. But one should never assume. Please take time to understand what your partner of choice is offering on retention policies, offloading backup data to a remote location, whether they have a remote datacenter at all and how they are dealing with connections between the datacenters.
If your partner of choice hase multiple datacenters, it may be possible to designate one of them as disaster recovery location. You will have the choice of leaving the Disaster Recovery into the hands of your hosting partner, or building your own solution for Disaster Recovery. Most of the time the hosting partner (or rather outsourcing partner) will, or should have expertise on building, maintaining and managing advanced topologies for Oracle E-Business Suite.

Disaster Recovery Establishing Oracle's Maximum Availability Architecture for E-Business Suite can be required to meet the availability and disaster recovery requirements. Your hosting partner should have multiple datacenters and have an idea about connectivity to the other datacenter when a disaster should occur. Creating MAA in one datacente doesn't make sense.

This all might seem trivial, but it is of greatest importance to cover before selecting your partner.

In the next article, I will discuss the surrounding systems, or supporting services.

Outsourcing your E-Business Suite environment - Part I

2009-02-11T08:11:00.008+01:00

Many, most smaller, companies do want to have all the benefits that the Oracle E-Business Suite can provide, however don't have the necessary resources to run, maintain and manage this complex suite of applications. A typical way to overcome these issues in general is to outsource this kind of solutions to a partner who is specialized in this matter. This can be done in various ways:

Remote Management
This way, the environment is kept inside the premises of the customer, but is managed remotely by a third party
Hosting
This way, the environment is hosted outside the premises of the customer, and optionally managed by a third party.

Hosting can be a good solution if you don't have enough resources to house your environment yourself, however a good deal of things need to be covered when selecting a hosting partner.

In the coming series of articles I will try to cover as many of these options as possible in order to make it easier to outsource your environment.

This first article will deal with the question why you would want to outsource.

The Oracle e-Business Suite is a complex suite of applications. It is composed out of several components, each requiring their own experience. For the bigger companies it may not be a problem hiring a team of professionals to maintain and manage this kind of environments. However, smaller companies often have an IT department that can do the daily management of the general IT infrastructure, but having a dedicated E-Business Suite DBA, or rather a couple of them (for continuity of service) can be too costly. Nevertheless, the E-Business Suite often is a business critical application.

Oracle E-Business Suite is a great application to be managed remotely. The architecture suits this approach seemlesly. It is designed to be implemented at large companies that often have multiple sites or even multinationals. Therefore, offshoring the E-Business Suite should not be a problem in itself, because it is designed to work in a multi site environment.

It is virtually impossible to understand, let alone master every aspect of the E-Business Suite, simply because it is too big a piece of software. That in itself implies that the E-Business Suite is applicable to almost any kind of business. One of the major advantages of outsourcing the E-Business Suite is that, if you choose the right partner, you can leverage the knowledge of all of your partner’s application administrators in order to ensure a smoothly running application in your organization. Be it Functional or Technical Application Administrators, they provide services for other companies as well, and consolidate this knowledge in their service offering. It is cost-efficient to outsource your environment, because you will only pay for the service you need or want. Besides, outsourcing partners have resources in place to maximize availability of service. Things you will have to pay for in full when you decide not to outsource.

Key factors of success when considering or selecting an outsourcing partner are trust, security, technical understanding, functional understanding, connectivity, continuity of service, and not to forget: performance. In addition to this, you need to understand yourself what your application is about. There is a whole lot of questions to ask your potential parters in order to get a good understanding of them being a good fit for you or not. All of these questions need answers. Only then you can make a solid choice whether to outsource and to whom.

In my next article(s) I will discuss some more of the technical questions you may need to consider for outsourcing.

Mixing E-Business Suite and Pleasure at Collaborate09

2009-01-14T19:34:00.002+01:00

Yesterday I received the long-awaited e-mail from the Oracle Applications User Group (OAUG) that my presentation submission called "Mixing E-Business Suite and Pleasure" has been accepted for Collaborate09. This will be the first time for me to present at such a large scale event.

For those of you who are interested, the presentation describes a project I have been involved with as DBA/Architect. This project was the largest Oracle E-Business Suite with SOA integration in the world at that time. The presentation covers the business case, the architecture, challenges and lessons learned.
Look for Session #1037!

I sure hope to meet many people in Orlando, it will be a great event!

Happy New Year!

2009-01-05T08:40:00.005+01:00

First of all, I would like to wish anyone out there a very happy and prosperous 2009!
For me, there is a number of challenges to resolve this year. I started at Inter Access on December 1st of last year, where I will be involved in the designing architectures and infrastuctures for Oracle E-Business Suite environments of our customers. We also provide a hosting solution for Oracle related solutions. I will be working in that area too. Hopefully I will be publishing a lot about this in the near future. I am sure there will be enough to write about.

EBS: No OCFS2 with Shared Application Tier File System

2008-12-22T09:20:00.003+01:00

Currently I am involved – sideways – in an Oracle E-Business Suite Release 12 implementation at one of our customers.
One of my colleagues is implementing a RAC solution with two nodes and two application servers. The platform is Oracle Enterprise Linux 4.
Before I joined, the team decided to choose OCFS2 as cluster file system for the database. A logical step would be to use OCFS2 for sharing the Application Tier File System as well.
However, we recently found out that this ORACLE product still is not supported/certified for use with the Application Tier of Oracle E-Business Suite. There are issues with OCFS2 in combination with a Shared Application Tier File System (see my article at the AMIS Technology Weblog). Somehow Oracle doesn’t seem to be putting too much efforts into certifying OCFS2 against the EBS Application Server Tier. Initially, the reason was that Oracle had to put too much work into Release 12 and Fusion. However, we are more than 2 years from the initial announcement by Steven Chan, and still no certification, even though customers are requesting it. The only workaround for the issue with OCFS2 in setting up a Shared Application Tier File System is the one proposed in my article, and it is this workaround being referred to by Oracle Support as well. But keep in mind that this doesn't mean it is supported, because it is not!
OCFS2 is a Cluster File System that is certified against and supported with Oracle Server. Why not with Application Server (EBS)? In my opinion there are many customers willing to use OCFS2, because it is stable, from Oracle and not at the least: free of charge.
By the way, the issue I found with OCFS2 also applied (at the time) to GPFS (IBM's Cluster File System). When I tried to reproduce it on AIX on Power, the issue did not reproduce. It might therefore be something related to Linux (whether it be OEL or RHEL) and not at all to OCFS2.
One thing is sure: I would want to use OCFS2 at my customers. I have enough of them who would be willing to use it.
Until that time, my recommendation is to stay away from OCFS2 with the EBS (shared) Application Tier. Use NFS, or any alternative which is supported instead.

1Z0-238 (EBS R12 IP&M) exam release date?

2008-12-16T11:41:00.004+01:00

On October 30 of this year I took the 1Z1-238 Oracle EBS Release 12 Install, Patch and Maintain Applications Exam. At the time it was still in Beta status, what is more, the Beta exam ended the very day after (Oct. 31). If you are interested in my experiences, please visit Mohan Dutt's weblog, which holds a blog from me sharing my experiences on the exam.
One of the nice features of Oracle's beta exams is that anyone who takes this exam will be scored on the basis of the actual production exam, resulting in a valid certificate for the production exam, when passed.
Like anyone else in my position would be, I am very eager to know whether I passed this exam. If I do, I will be among the first to get the OCP status for E-Business Suite Release 12, because I already have got the 11i OCP certificate.

Today I was checking the prometric website to see whether the scoring had been applied to the test. Unfortunately, the status still was "Tested"... To get an idea of when to expect the scoring result, there is Oracle's statement that it will take approximately 8 to 10 weeks after closure of the Beta Period, but you can always check with Prometric what the availability is for the actual production exam. So I looked up 1Z0-238 (beta exams always start with 1Z1, production exams start with 1Z0) and started the scheduling engine for this exam. This way I found out that the first available date for the exam is set to February 2nd (strangely enough, the exam can only be scheduled until February 10?).

Anyway, I guess my results will be made available somewhere in January... I need some more patience...

UPDATE, January 6, 2009:
I was checking again today and saw that the first exam-date for this exam is now scheduled at January 20, 2009.

Recovering Large Databases In Shortest Time Possible

2008-11-13T08:00:00.009+01:00

I have published this blog before, at the AMIS Technology Weblog:

Sometimes you come across beautiful things in your life, you wish you had known before, because they were there all the time, waiting for that moment to be discovered – by you.
Recently I discovered one of these things. Shamefully I confess, this brilliant feature of RMAN is around for some time – since 10g already – but I have never been lucky enough to explore it, until recently.
I was working for a customer, who has a number of very large databases, all of them around the size of 12 to 25 Terabytes. These databases are all very heavily used by all kinds of activities; each of them is producing over 150 Gigabytes of redo per day. Making a full online backup (to disk) of these databases takes more than a week – per database…
Can you imagine what it takes to restore and recover this kind of databases? The customer estimates around 2 weeks. This is not an option, because these databases hold data that is very frequently consulted and processed.
So, what to do in such a case? Well, RMAN has a nice feature called backup as copy. This makes an image copy of your database and puts it into another location, defaulting to the Flash Recovery Area (assuming you have set up your parameters DB_RECOVERY_FILE_DEST and DB_RECOVERY_FILE_DEST_SIZE). I knew about this feature because I have migrated lots of databases from file system to ASM, for example.
What I didn’t know is that you can use incremental backups to recover this copy of the database. A jewel in the RMAN feature portfolio!
To make a full copy backup of your database:

allocate channel for maintenance type disk;
configure controlfile autobackup on;
configure default device type to disk;
run
{
RECOVER COPY OF DATABASE WITH TAG ‘IMG_COPY’ UNTIL TIME ‘SYSDATE – 1’;
BACKUP INCREMENTAL LEVEL 1 FOR RECOVER OF COPY WITH TAG ‘IMG_COPY’ DATABASE PLUS ARCHIVELOG DELETE INPUT;
copy current controlfile to ‘/path_to/cp_cntrl.ctl’;
}

OK, let’s break this down into small pieces:

RECOVER COPY OF DATABASE WITH TAG ‘IMG_COPY’ UNTIL TIME ‘SYSDATE – 1’;

This command will not do anything until there is a backupset with the tag FULL_COPY.
When there is a backupset, it will take it and recover the copy of the database
using this backupset (see next command).

BACKUP INCREMENTAL LEVEL 1 FOR
RECOVER OF COPY WITH TAG ‘IMG_COPY’ DATABASE
PLUS ARCHIVELOG DELETE INPUT;

This command will create a level 1 backup if there is a level 0 available. During the first run of this script, there is no level 0, so the level 0 will be created (which is basically equal to a backup as copy database).During the second run of the script, a level 1 incremental backup will be created, which can be used in effect for the recover command as discussed above.
Basically, you are creating an image backup of your database in the Flash Recovery Area. This image backup can be used for restore and media recovery. Since this will be faster than conventional backups, it could be a solution to look at when you demand quick recovery. Well, nothing spectacular, this solution has been discussed a number of times on the Internet already.
In the situation my customer is in, we’re talking different stuff. The database is so large, taking the initial backup takes more than a week, let alone a restore, and we’re not even talking about recovery yet. So we chose a different approach to the ‘recovery’ of the database in case it becomes damaged.
RMAN has this tiny little command that does it all in this case:

RMAN> SWITCH DATABASE TO COPY;

This only works when you have your controlfile available, but hey, that is what we do twice in the backup script, don’t we? And not for nothing.

copy current controlfile to ‘//copy_control.ctl’;

So, after the backup has been taken, a copy of the controlfile has been made to a separate location, eliminating the need to restore it from autobackup with all of its consequences. If somehow we have lost this image backup of the controlfile as well, we can always fall back to the autobackup.
When we want to recover our lost database in the least amount of time, we assure the availability of the controlfile, then switch the database to the copy, which in effect updates the controlfile with the new location of all the datafiles. After this has been completed, we can open the database (with or without resetlogs, depending on incomplete or complete recovery), and off we go.
The database is available now, but you will be left with a database that doesn’t have a valid backup anymore. We’re using the database in the FRA, remember? No panic, redefine your FRA (to e.g. the location of the original database) and start your rman scripts to create a new copy. Optionally, if required, you can relocate your datafiles according to your needs.
The downside of this solution is that you would need twice the amount of diskspace for your database, because you are holding a copy of it on disk. But compare the cost of that against the cost of 2 weeks inavailability of the database. I think in most cases (with current storage costs) the costs will be far less than the benefits.

How to clone an Oracle 10g ORACLE_HOME easily

2008-11-12T13:26:00.009+01:00

Last Friday I had been working very hard to install Oracle 10g software on no less than 8 servers. This included 1 ORACLE_HOME for ASM, and 2 separate ORACLE_HOMEs per machine for the separated databases to be installed and configured. So, in total a number of 24 installations of 10.2.0.1 and on top of that 24 times the 10.2.0.3 patch set.There should be an easier way to do this, I thought. With my Oracle E-Business Suite background, I know it is possible to clone an RDBMS ORACLE_HOME, but EBS uses specific scripts for this procedure, and this was not EBS; just plain Oracle Database Software.
I started googling around, and found out there is a perl script, delivered with Oracle 10g, to clone the ORACLE_HOME. It is located in $ORACLE_HOME/clone/bin (where have I seen this kind of directories earlier..?). It turns out to be really easy to clone an ORACLE_HOME.
Here are the steps, including the expected output:

1. Copy the Oracle_HOME to the destination location (whether it be on the same or a remote server). When you need to rename the directory where the current installation is located, you can do this as well. If necessary, take the central inventory as well.

2. Check the existence of the ORACLE_HOME in the Central Inventory

$ pwd/appora/oraInventory/ContentsXML$ grep
"HOME NAME" *inventory.xml:
$

3. (Conditional) Detach the ORACLE_HOME using runInstaller from the copied ORACLE_HOME

$ $ORACLE_HOME/oui/bin/runInstaller
-detachhome ORACLE_HOME=/appora/product/10.2.0/db
Starting Oracle Universal
Installer...

No pre-requisite checks found in oraparam.ini, no
system pre-requisite checks will be executed.
The inventory pointer is located at
/var/opt/oracle/oraInst.loc
The inventory is located at
/appora/oraInventory'
DetachHome' was successful.
$

4. Check the Inventory to see if the ORACLE_HOME was removed (verify the REMOVED=”T” option is added to the ‘HOME NAME’ tag)

$ cd /appora/oraInventory/ContentsXML
$ grep "HOME NAME" *
inventory.xml:
$

5. Reregister the ORACLE_HOME
Make sure nothing by the current user is running, because it will do a relink as part of the procedure.

$ cd $ORACLE_HOME/clone/bin
$ perl clone.pl ORACLE_HOME="/appora/product/10.2.0/db" ORACLE_HOME_NAME="OraDb10g_home1"
./runInstaller -silent -clone -waitForCompletion "ORACLE_HOME=/appora/product/10.2.0/db"
"ORACLE_HOME_NAME=OraDb10g_home1" -noConfig -nowait
Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2008-11-07_03-21-26PM.
Please wait ...Oracle Universal Installer, Version 10.2.0.3.0 Production

Copyright (C) 1999, 2006, Oracle. All rights reserved.

You can find a log of this install session at:
/appora/oraInventory/logs/cloneActions2008-11-07_03-21-26PM.log
....................................................................................................
100% Done.
Installation in progress (Fri Nov 07 15:21:37 CET 2008)
.................................................................................
81% Done.
Install successful
Linking in progress (Fri Nov 07 15:21:49 CET 2008)
Link successful
Setup in progress (Fri Nov 07 15:22:56 CET 2008)
Setup successful
End of install phases.(Fri Nov 07 15:23:03 CET 2008)
WARNING:
The following configuration
scripts need to be executed as the "root" user.
#!/bin/sh
#Root script to run
/appora/product/10.2.0/db/root.sh
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts
The cloning of OraDb10g_home1 was successful.
Please check '/appora/oraInventory/logs/cloneActions2008-11-07_03-21-26PM.log' for more
details.
$

6. Run Root.sh

$ su -
Password:
#
# /appora/product/10.2.0/db/root.sh
Running Oracle10 root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /appora/product/10.2.0/db
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
Copying coraenv to /usr/local/bin
...
Entries will be added to the /var/opt/oracle/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
#

7. The clone is ready now.

The nice thing about this is that once you have an ORACLE_HOME with a number of patches (like I had with the 10.2.0.3 patch set) these will automatically be taken with the clone.

Day 1

2008-11-12T10:30:00.007+01:00

Another Oracle related weblog is born today. I have been struggling with the idea for months, if not years, and never took the time to create one. Now finally the time has come.

Well, what is the purpose of this weblog? The idea is to share my knowledge with the community; it'll also have to serve as some kind of reference for the future.
The majority of the articles here will be focusing on the technology or technicalities of Oracle E-Business Suite and Oracle Database.

Some Information about myself can be found in my profile as well as on LinkedIn.
I hope to be a frequent blogger. I have published some weblogs at the AMIS weblog, and will continue to do so here. Some of the posts that I published at the AMIS weblog will be published here as well in the near future.

Hope to meet you here again soon!