Community Server for Rent a Coder

How Software Gets Done
Welcome to Community Server for Rent a Coder Sign in | Join | Help
in Search

Real Life Microsoft IT

SQL Server 2005 Service Pack 1: The good news and the bad news

SQL Server 2005 Service Pack 1 (SP1)includes many bug fixes and improvements to the core SQL Server 2005 engine.  Given the poor performance we had been experiencing after upgrading from SQL Server 2000 (despite simultaneously doubling the processors, tripling the memory, increasing hard drive speed, etc.) I was excited to try this out.  This quote from the above URL looked especially promising:

"Several performance problems were fixed in the Database Engine query optimizer. These changes improve the upgrade process for third-party products that use SQL Server as their back-end database when you upgrade from SQL Server 2000 to SQL Server 2005."

What happened?

The good:

Via an off-the cuff estimate, it appears that things are indeed running slightly better.  The processor was averaging around 92% CPU usage on Monday and dropped to 84% today after installation of SP1. 

However, when put into perspective...the old SQL 2000 server (with half the machine, 1/3 the memory etc) was running at 80-90% CPU.  So this is just one small step in the final journey.  But nonetheless...it's a step in the right direction.

The bad/ugly:

Now for the bad news.  After installing SP1 I thought everything worked fine.  Today there were some problems with a replicated table.  After probing here is what greeted me in replication monitor:

A complete 100% failure of all replication on every single publiccation and subscription to every single database.  I didn't realize right away, because database mail was not working properly pre SP1 on x64 machines.  There was a KB article on how to get around this bug, but it didn't work (I had a ticket open for it).  Now it's a moot point because with SP1 it's been fixed!  Yes...I will now enjoy plenty of warning when future service packs break things. Goody! :)

On this one it's tricky to figure out what the cause is.  The normal history and log menus are completely disabled in replication monitor so I can't even see what the error/problem is.  Arrrghh.  Time to open another ticket...sigh.  (I opened 3 previously today on SQL Server...and it's only 11:19PM so that makes 4 in one day.   Can someone remind me again why I'm paying Microsoft for this product and not the other way around?)

4/27/2006: Update on the replication issue: I got away without opening a ticket (after getting some rest and thinking about it).  I noticed that SQL Server lost the names of the replication databases (for some reason) on each subscription, after SP1 was installed.  With some experimentation, I found there was a "simple"workaround...blowing away all publications and subscriptions and recreating them from scratch. 

Simply wiping out just the subscriptions and redoing them would have been easier, but that didn't completly work.  The new subscriptions functioned, but for some reason the old ones (with missing database names) were still there, like "phantoms".  Leaving it this way would have made the replication monitor useless, because it would have always indicated an error on every publication (for the phantom subscriptions)..even when the "real" ones were fine.

So fixing it required blowing away the subscriptions as well.   This is not a big deal if you only have a few tables that are small.  Fortunately that's our case. Unfortunately one of them allows the sharing of information on site members across the Exhedra sites (and databases).  This big guy is a few million rows. So this meant resnapshotting and sending all that info over the network to the other databases again.  The users on those sites are not very happy with us right now, becuase this is killing their activity on the databases/sites.  But it's one of those things that "has to be done".  Otherwise no one could create new accounts...not a good situation either.

So my recommendation is to allocate extra time in your Service Pack1 Upgrade planned downtime to possibly handle rebuilding all of your replication. Good luck.

Ian Ippolito

Published Wednesday, April 26, 2006 11:01 PM by admin

Comments

 

arbert said:

Personally, instead of paying 10k for Microsoft "support", I would have found a local consultant that has some know-how.

I've yet to install any Sp for SQL Server that didn't affect replication in some weird way.  I'm really amazed with the problems you're having--we haven't had many issues with our installs.  The largest install we're running right now is about 400gig database size, 32gig  ram, 4procs.....I will say, you're definately going to get a bigger bang by upgrading memory than you will by upgrading processors--16gig isn't a whole lot--especially for a 64bit box.

Do you guys publish any of your technical specs of the database, I'd be interesting in how your stuff is laid out--always neat to see what others are doing with SQL.
April 26, 2006 11:47 PM
 

Ramblings of a DBA said:

April 26, 2006 11:50 PM
 

phpgeekster said:

What kind of challenge would it be to rebuild RAC on Linux, Apache, MySQL & PHP (LAMP)?

I'd love to work on that kind of a project! ;-)
April 27, 2006 12:25 AM
 

redbrad0 said:

I sent a message to the admin because we also used to fight with SQL Server all the time. We switched over to mySql and you wouldnt believe the better response we are getting. All of our pages are ASP and it did take some work but 95% of all the code worked with mySql. All "select top xxx" had to be changed to "limit xxx", handeling of dates (2006-04-27 instead of 4/27/2006) and then a couple number problems but other then that we are glad we moved.

We current have approx 5 million articles that get searched a couple million times a day. SQL Server full text would return results in approx 10 seconds so we purchsed SQL  Turbo which lowered it down to about 1 second but we kept having connection limit problems. Now we return results in 0.02 seconds on avg searching 5 million + querys a day.

RentACoder... please look at switching to mySql I will help you the entire way.
April 27, 2006 11:51 PM
 

GregAndTania said:

I think instead of rebuilding RAC on Linux it shall be rebuild based on .NET 2.0, it's time to move in 21st century
April 28, 2006 7:38 AM
 

phpgeekster said:

# RentACoder... please look at switching to mySql I will help you the entire way.

I say they focus on a complese revisioning, but without a glimpse at what they're up against, it's hard to know how to fix the problems. $10k @ MS might work better for now than what I have in mind, although every single feature could be built into a PHP/MySQL version of RAC for a FRACTION of that support price. ;-)

Then we could start talking about some really funky community features. I would like to be able to moderate bids. ;-)

ie: "SEO help me pleeze? 20 articles, 300+ words..." ($35.00 Max) (Score: -1, Slave Troll)
or:
"Help Anne Rice write a novel!" (Open to Suggestions) (Score: 5, Hot Project)

Cheers!
Scotty
April 28, 2006 10:43 PM
 

admin said:

I'm pleased to announce that we've resolved 95% of the SQL Server database problems and CPU and everything else is now running great!  For full details see:
http://rentacoder.com/CS/blogs/real_life_it/archive/2006/04/28/477.aspx

To answer some of the questions above:

1) Memory: Performance monitor and SQL Server weren't demonstrating that memory was a bottleneck or a problem (other than when pressured by the backups, which is a seperate blog issue).  So upgrading simply to up the amount would not have gotten us anything.

2) Complete rewrite: Despite the "regligous" excitement that the prospect might raise for some people :) ...this is actually something that would be an absolute LAST resort due to the drastic nature of it.  Fortuantely an easier, less costly and less time consuming solution was dicovered.

3) .NET: There is another blog posting on this issue, but basically forward is the only way to go.  We'll be upgrading piecemeal to ASP.NET once the Service Pack on the "edit and continue" release has matured the product (we've learned what can happen to early adapters pre Service Pack 1 on this issue).

Ian
May 1, 2006 8:32 PM
Anonymous comments are disabled

This Blog

Post Calendar

<April 2006>
SuMoTuWeThFrSa
2627282930311
2345678
9101112131415
16171819202122
23242526272829
30123456

Post Categories

Syndication

Powered by Community Server, by Telligent Systems