My Daybook ~ March 2

 Outside My Window…there are limbs from our orange tree scattered across the backyard. I began to ‘excavate’ my ’sacred/happy’ place in the backyard this weekend.

I am thankful….for the insight being provided by my re-reading of ‘Simple Abundance’.

From the kitchen…actually quite a bit has been going on in there. I did make the baked spaghetti. It was awesome!!! Then I made a new recipe I found at Mennonite Girls Can Cook for meatloaf. It was out of this world delicious. And of course it made THE BEST meatloaf sandwiches the next day.

Baked Spaghetti

16 oz. spaghetti noodles, cooked

2 (24 oz ) jars Ragu Traditional spaghetti sauce

2 lobs. ground beef, browned

1/4 cup butter

1/4 cup all purpose flour

1/4 cup grated (fresh) Parmesan cheese

1 pkg Good Seasons Zesty salad dressing dry mix

12 oz. evaporated milk

3 cups shredded sharp cheddar cheese, divided

Combine spaghetti noodles, spaghetti sauce and ground beef in a large bowl; set aside. Melt butter in a sauce pan over medium heat; add flour, Parmesan cheese, Good Seasons mix, stirring constantly until smooth and bubbly. Add evaporated milk and one cup cheddar cheese; stir until thickened.

Pour half of spaghetti noodle mixture into a greased 13 x 9 casserole dish and pour the cheese mixture over the top. Pour remaining noodle mixture into dish; top with remaining 2 cups of cheddar cheese.

Bake at 350 degrees for 25 - 30 minutes. Serves 12.

By Far The Best Meatloaf

Ingredients:

1 1/4 lbs lean hamburger

1 tbsp broiled steak seasoning

1/2 cup dried bread crumbs

1/2 cup rolled oats

1/4 cup ketchup

1 cup grated cheese

1/3 cup milk

Topping:

1/4 ketchup

1/4 cup BBQ sauce

2 Tbsp brown sugar

Mix all ingredients well and shape into loaf. Place into a 9 x 13 baking pan and make an indentation along the top of the loaf. Cover meatloaf completely with topping. Bake at 350 degrees for about 60 75 minutes depending on the shape.

I actually doubled the recipe so I would have some left over for sandwiches.

 I am creating…an updated the original of hair extensionsHousehold Notebook.

From the learning rooms…definitely absolutely nothing going on there. Miss Sam was so completely under the weather for over a week that she couldn’t even read (her eyes hurt so bad). Today we hope to get back into the swing of things…but still taking baby steps as her energy level is still not up to par. So today it will be reading her lessons and math. I took out grammar and vocabulary today.

I am going to…have an early dinner with a colleague today before heading to class tonight.

I am reading…Inkheart by Cornelia Funke. I am really enjoying it. I must admit however that it does get a little bogged down in details so I do ’scan’ quite a bit. However, the story line is interesting and the characters are quite engaging.

I am hoping…to get my taxes done today. I made an attempt yesterday but our Brighthouse server was having mega problems due to the high winds we were experiencing. Thankfully we didn’t experience any of the snow that many of my blogging friends are dealing with today.

Around the house..I got rid of another chair in the front living room and replaced it with a live plant. Now when you walk into our home it finally appears as spacious as it did the first time I walked into it and fell in love with it. I’m starting to set the plans in motion for the ‘re-do’ of our bedroom.

One of My Favorite Things…is hooking up with blogging friends and in real life friends through Facebook. It has become an awesome tool in re-connecting and staying cthe tips of washing duvet coversonnected with people who make my world a happier place.

A Few Plans For The Rest Of The Week: catch up from last week.

Here is a picture thought I am sharing…

 

The beginnings of my ’sacred/happy’ place. There is still quite a bit of raking to do.

Today’s ‘Around the Square‘: is Organized Home. Not only do they have a vast amount of resources to peruse as we all head into a spring cleaning frenzy. I also love (and use) the printabthe suggestion of washing duvet coversles they have available. If you need to feel inspired with spring around the corner…go over there and take a peek at all there great suggestions for getting and keeping your home organized.

Today I am grateful for: the flexibility of my counseling job.

Today I am going to simplify my life: by updating all the information in my Household Notebook and adding additional info that I am usually on the hunt for….like non-reimbursed medical expense sheets that I need for my taxes!!!!

Samantha’s Corner:

Dear Sam,

It’s 10.23 in the morning and you are still sleeping. I never thought I would be a parent that would indulge this type of behavior. However, I never thought that I would be the parent of a child whose internal clock is set in ‘night owl’ mode. Ever since the day you were born, getting you to sleep before midnight has ALWAYS been a struggle. And so what kept you up until all hours of the night last night???

Drawing this:

 

However, it is time to get up and get started on your day even though your ‘muse’ keeps you awake into the wee hours of the morning. Perhaps your ‘muse’ can help with your school work????

Love, Mom

Share This

Using Decision Automation in Disaster Recovery

DM Review recently published my article on “Using Decision Automation in Database Administration” (http://www.datawarehouse.com/article/?articleId=6876). Right after that, I get this interesting question from a reader enquiring about leveraging decision automation in disaster recovery (DR). Specifically, she asks “a good application I can think of for decision automation is disaster recovery. Can you outline a model for how one could go about implementing it?”. I wanted to share my response with everyone. So here I go…

Once the right DR model is designed, automation in general, icurly hair extensionss invaluable in rolling it out and maintaining it; ensuring it works as per spec ("spec" being the enterprise service level requirements pertaining to uptime, failover and performance). However the one area that decision automation specifically is suited for is establishing a centralized command / control mechanism to allow failover to occur under the right circumstances, and in the process, dealing with false alarms.

The foremost (and most natural) reaction of many IT administrators when they face an outage (even a simple disk-array failure, much less a true disaster) is panic. During this state, the last thing you want them to do is think on their feet to figure out whether to fail over to the DR system and if so, go about making that happen manually. Especially, if there are other unanticipated hiccups in the process (which happen more often than not at that very time, given Murphy's law), that usually results in service level violations or worse, a failed DR initiative.

To better appreciate this, imagine a nation’s nuclear arsenal being subject to manual controls and individual whims rather than a fully automated and centralized command / control system. If relying on a human being (be it the president or prime minister or your local fanatic dictator) or worse, multiple humans to make the call re: launch and then deploy manually, the system would be prone to mood swings , and other assorted emotions and errors resulting in the weapons being deployed prematurely, or not at all even when the situation calls for it. Further, once the decision is made to deploy or not deploy, manual errors can delay or prevent the required outcome.

These are the situations where decision automation can come to the rescue. Decision automation can completely do away with the manual aspects of deciding whether or not to initiate failover and how (partial or full). It can coldly look at the facts it has accumulated and allow pre-determined business logic to figure out next steps and initiate an automated response sequence as required. Let's look at what that means.

Let's say the service level agreement requires a particular system or database to provide 99.999% uptime (five 9s). That means, that database can be down for a maximum of 5 minutes of unplanned downtime during the year.

Let’s make thair extensionshe problem even more interesting. Let's pretend the company is somewhat cash-poor. Instead of spending gobs of money on geo-mirroring solutions like EMC SRDF and other hardware/network level replication (even “point-in-time copy” generators like EMC’s SnapView, TimeFinder or NetApp’s SnapVault), it has invested in a simple 2-node Linux cluster to act as the primary system and two other SMP machines to host a couple of different standby databases. The first standby database is kept 15 minutes behind the primary database and the other standby remains 30 minutes behind the primary. (The reason for the additional delay in the latter case is to provide a longer window of time to identify and stop any user errors or logical corruption from being propagated to the second standby.) The primary cluster is hosted in one location (say, Denver) and the other two standby databases are being hosted in geographically disparate data centers with data propagation occurring over a WAN.

Given this scenario, at least two different types of failover options emerge:
- Failover from the Primary cluster to the first standby
- Failover from the Primary to the second standby

The underlying mechanisms to initiate and process each option is different as well. Also, given that the failover occurs from two-node clustered environment to a single node, there may be a service level impact that needs to be considered as well. But first things first!

Rolling out this DR infrastructure should be planned, standardized, centralized and automated (not “scripted”, but automated via a robust run book automation platform). You may think, why the heck not just do it manually? Because building any DR infrastructure, no matter how simplistic or arcane it is, is never a one-time thing! (Not unlike most IT tasks…). Once failover occurs, failback has to follow eventually. The failover/failback cycle has to be repeated multiple times during fire-drills as well as during an actual failure scenario. Once failover to a secondary database happens, the primary database has to be rebuilt and restored to become a secondary system itself or in the above situation since it's a cluster (presumably provides more horsepower and the system when it fails over to a single node has to operate in reduced capacity), failback has to happen to reinstate the cluster as the primary server. So in the frequent cycles of failover, rebuilding and failback, you don't want people to keep doing all these tasks manually. The inconsistency in quality of work (depending on which DBA is doing which task) could cause inethe original of hair extensionsfficiencies to creep in resulting in a flawed DR infrastructure. Human errors could compromise the validity of the infrastructure.

So the entire process of building, testing, maintaining, and auditing the DR infrastructure needs to be standardized and automated. Also included in the “maintenance” part is the extraction of the archived/transaction logs every 15 minutes, propagating to the first standby server and applying it. Similarly, in the case of the second standby, the logs have to be applied every 30 minutes to ensure the fixed latency is maintained. In the case of certain versions of certain database platforms (namely, Oracle and DB2), there are DBMS supplied options to establish the same. Regardless, the right mechanisms and components have to be chosen and deployed in a standardized and automated manner. Period.

Now, the DR infrastructure could be well deployed, tested and humming, but during the critical time of failover, it needs to be evaluated what kind of failure is being experienced, what is the impact to the business, how best to contain the problem symptoms, transition to a safe system and to deal with the impact on performance levels. Now, applying that to our example, it needs to determined:
- Is the problem contained in a single node of the cluster. To use an Oracle example, if it’s an instance crash, then services (including existing connections, in some cases such as via Transparent Application Failover or TAF) can be smoothly migrated to the other cluster node.

- If the database itself is unavailable due to the shared file-system having crashed ( in spite of its RAID configuration), then the cluster has to be abandoned and services need to be transferred to the first standby (the one that’s 15 mins behind). As part of the transition, any newer archived logs files that haven’t yet been applied need to be copied and applied to the first standby. If parts of the filesystem that hold the online redo logs are not impacted, they need to be archived as well to extract the most current transaction(s), copied over and applied. Once the standby is synchronized to the fullest extent possible, it needs to be brought up along with any related database and application services such as the listener process. Applications need to be rerouted to the first standby (which is now the primary database) either via implicit rerouting by reconfiguring any middleware or via explicit rerouting (caused by choosing a different application configuration file/address or even, lower level mechanisms such as IP address takeover, wherein the standby server will take over the public IP address of the cluster).

- If for any reason, the first standby is not reliable (say, logical corruption has spread to that server as well or it is not starting up in the manner expected due to other problems), the decision needs to be made to go to the second standby, carry out the above described process and bring up the necessary services.

The real challenge is not just having to make all these decisions on the fly based on the type and scope of failure, and the state of the different primary and standby systems , it is having to make them and complete the entire failover process within the required time (5 minutes, in our example). Organizations cannot realistically expect an panic stricken human being to carry out all this quickly on the fly (yet, ironically, that’s exactly what happens 80% of the time!).

IT professionals sometimes say, “I don’t trust automation; I need to be able to do things myself to be able to trust it.” Well, would YOU entrust such a delicate process wrought with constant analyzing/re-analyzing and ad-hoc decision making to a human being who may be sick or on vacation and out of the office? Someone who may have left the company since then? Would you be content merely allowing this process to be documented and hoping someone else (other than the person who designed and built the solution) in the IT team can perform it without problems? I for one, wouldn’t.

The range of situations for applying decision automation are manifold and implementing a centralized command / control system for initiating failover in the DR process just happens to be an ideal candidate.