Yet another SharePoint BLOG

Some time ago, one of our customers reported an odd problem. The support ticket went something like this:

When a user tries to create an alert there is a long pause and is then prompted to login. They enter their credentials but they then receive the following error: "you are not authorised to view this page.

It’s taken a long time to get to the bottom of this one due to a number of red herrings so feel free to skip to the fix at the bottom.

The site in question had recently been moved from another set of servers. The site owners had taken the opportunity to rebrand the site and so our immediate investigations were directed toward the new farm and those customisations.

It quickly became clear that it was a subset of lists in the site that were effected but there was no obvious pattern between them.The "bad lists" included document libraries, news lists and discussion forums and it wasn’t anything simple like the lists with lots of items or a particular file type.

Not long after we started looking into this problem we started having occasional IIS application pool crashes. This obviously took priority so efforts were diverted. To be honest, we weren’t making much progress with either issue until one of our contractors noticed that reproducing the alerts problem triggered a hang on the application pool. If enough people tried to create an alert the application pool would enter a stopped state.

Goggling “A process serving application pool * exceeded time” brings back lots of generic IIS blogs and forums, most of which point to this post on the IIS team blog which describes using DebugDiag to isolate the issue. The dumps pointed us towards an issue with InfoPath forms and in turn this forum post but again this turned out to be a dead end.

The Solution

So after a lot of digging we were finally able to work out that the effected lists all had rather complicated views. To give you an example, one of these lists had 64 views, one of which had 12 filters, i.e. column A = XXX AND column B = YYY etc. Deleting that complicated view allowed the user to create alerts.

What’s happening is that the filter creates a CAML query. When that query gets above a certain length (which appears to be about 500 characters) it hangs the application pool. If enough people try the query at the same time, the application pool crashes. To the user this manifests as a long pause followed by being prompted for credentials and then getting “access denied” On the server the application pool is rebooting and reaching and then reauthenticating the user.

So with all this information in hand we logged a fault with Microsoft and they’ve accepted it as a fault. They expect the fix to be included in SP2.

I’ve found myself working with LiveMeeting a lot recently and so have more LM qwerks to discuss than SharePoint ones. The latest one, I’m pleased to say, has been accepted by Microsoft and will be added to the next cumulative fix. The jist is that a window changing form is recognised as a new window by the presenters client and as a result, the content is no longer shared. Below is a worked example

Open Calculator and share it

What you should see is the calculator with “currently sharing” in the title bar and the rest of the screen partially greyed out.

Now switch to scientific mode and the presenter and the viewer will be kicked back to the “Nothing is currently being shared” screen.

For clarity, this isn’t restricted to calculator but it makes for a simple example. The problem is that in 2007 the product team have locked down the window security (this doesn’t happen in LiveMeeting 2005) and so when changing to scientific mode, LiveMeeting considers it a new window and so closes the sharing session. We were using this as a remote training tool for new starters. Often, trainers didn’t notice that the application wasn’t being shared and the new starter was too nervous to say anything and so sat there looking at a grey screen for an hour!

Microsoft’s current work arounds are:

• Share the entire desktop (you can’t hand over control in this configuration)
• Open a remote desktop session to another computer with calculator running and share the RDP client
• Create a virtual PC running calculator and share the VPC

I’ll post an update once Microsoft release a fix.

Ok, this isn’t SharePoint related but hopefully it’ll be helpful to someone. We recently rolled out Livemeeting 2007 across the company. It turns out the client does a version check at the end of each meeting and will prompt the user to update.

Obviously you’ll want to suppress this for users who don’t have admin access. Create the following registry key on the client machine and set the value to 1

HKEY_LOCAL_MACHINE\Software\Microsoft\ Live Meeting\8.0\Lockdown

This and other useful reg keys (particularly in relation to OCS) are available here

A colleague of mine recently blogged about the difficulties of a managing SharePoint environment with a mixture of standard users enterprise. In short, he came to the conclusion that it isn’t a viable solution – a conclusion that I’d whole heartedly support given the grief it causes us both each day. If I had my time over I wouldn’t have attempted it. if you still want to try the mixed environment after you’ve read this post I’d encourage you to have a read of as it will help give you some pointers.

Paul’s post prompted somebody from Unilever to contact him as they’re currently in discussions with Microsoft and were having difficulties getting a simple answer as to the implications of standard vs enterprise. I remember those discussions well and the time I spent trying to find a clear answer online – there isn’t one. This is my attempt to clear the confusion.

Everything below is based on my experience working for a big company with a fair bit of buying power. I don’t know how the numbers will translate to your company or how Microsoft will behave if you get it wrong. With that in mind, do as much research as physically possible because if you get it wrong and Microsoft find out, they’re well within their rights to penalise you.

Server and user licenses

SharePoint is licensed in two parts. The server part is the simplest so lets start there.

Although you can have either an enterprise or standard server, there is no difference in cost (we pay a little under £2’500 + the windows license per server.) The license simple enables a number of features for the farm. You cannot have a mixture of standard and enterprise servers in the same farm. You can upgrade a standard farm but you cannot downgrade an enterprise farm. There is no definitive wording from Microsoft but I’ve seen it argued that if you have an enterprise farm in your company, every single user must have an enterprise Client Access License (see below for more info on CALs.)

Client Access Licenses (CALs)

In order to access a SharePoint site you should be covered under a CAL. Our site gives all of our users access to Windows, Office, Exchange, AD, DNS and SharePoint Standard so if our servers were running the standard license, we’d be fully compliant. As soon as an enterprise feature is enabled on a farm, web application, site collection or site that a user can access, that user requires an enterprise CAL which costs about £45 each. My company has around 80’000 users so in theory, simply turning the farm on meant they could access it (note that they don’t actually have to access it, only be able to) and required us to pay Microsoft £3’600’000 for the CALs.

Now I’ll quickly point out that we didn’t fully understand the implications of enterprise vs standard CALs when we installed SharePoint and ran none compliant for about a year. When we realised the mistake we spoke with Microsoft who were very understanding (I’m guessing it happens a lot.) They entered into an agreement with us where we report on the users accessing enterprise enabled sites and pay for them. Microsoft are under no obligation to agree to this kind of payment model and I doubt they would do it for smaller companies. It’s also worth pointing out that SharePoint doesn’t offer this level of reporting. I had to code an application which scans the farm looking for sites with the enterprise features turned on then grabs the user list and merges it into a spreadsheet. I’d originally planned on using an event receiver to hook when an admin turns features on and kick off some workflow which charged the appropriate business area, but it turns out no event is triggered.

Internet/extranet

If you have an extranet or internet based on SharePoint you’ll need another license called “Microsoft Office SharePoint Server 2007 for Internet sites” and if you run a farm with internal and external users you’ll need both licenses. We don’t use SharePoint externally so I don’t have any numbers to offer you. A quick Google shows that the prices appears to be around $40’000

What does enterprise give me

I’m quite critical of the way Microsoft handles standard vs enterprise licenses but to be fair to them, they’re very clear on what each one gives you (assuming you know where to look.) Start here. The short version is that if you want to use Business Data Catalogue(BDC), Forms Services, Excel Services or Report Centre you’re going to have to put your hand in your pocket. Keep in mind that while in most instances your farm admins will have to do some work to enable them, your site admins can still try to use web parts, making their site enterprise. Basically you’d be paying for something that doesn’t work. Clearly education is key here and needs to be embedded firmly into your governance.

Mauro Cardarelli has done a pretty good job of expanding on what enterprise gives you versus standard.

Every business case for a technical solution looks something like this:

We will save the business money by streamlining process/reducing costs/reducing headcount/driving efficiencies/realising business benefit [delete as appropriate.] This will be achieved by the delivery of a integrated system based on [insert the technology of choice.]

Benefits of £xxxxx will be returned to the business.

So in the the current climate of cost cutting and emphasising profitable sections of the business, why aren’t businesses investing like mad in collaborative technologies such SharePoint, Notes, WebSphere etc? Either the business is short sighted or the business case was less than genuine…

Today hasn’t been a great day:

• I spent 45 minutes stood at the station only to find out that one inch of snow in enough to stop a train. Trains which of course run four inch above the ground on iron tracks.
• I’m in the middle of the project from hell where some developers have thrown WSS, Biztalk, K2 and a shed load of custom code into a mixing pot without any consideration to how it’ll work in production (“it works on our dev server” – “your dev server is a single box with everything running under one admin account” – “so?”) or how it’ll be supported.
• My uncle got taken to hospital so I’ve spent all evening with my grandma holding back the tears.
• My puppy left me a present in the kitchen.

So why am I taking the time to write a post after such a rubbish day? Because I finally got to put a six month old problem to bed.

“I cannot edit the properties or text of the first post in a discussion forum. When I click edit I get ’an unknown error has occurred’. I can edit all other posts.“

When this bug was first reported our support guys went through the usual process of checking ULS logs, event logs etc but everything was clear. This was the only one of our four farms to show the behaviour and Google was disappointingly unhelpful. We were left with little choice but to engage PSS. More log checks, reproducing the error (which Microsoft were finding hard), SQL traces – you name it, but almost six months later and it appeared MS were as stumped as we were.

Then today, out of the blue they fired over an e-mail to say it was a bug that’d be introduced with SP1 and fixed with the infrastructure update.(we’d put fixes on hold while we ironed out some of the outstanding PSS cases.) I installed it into a dev system and sure enough, the problem went away.

It’s slightly embarrassing to say I sat on a bug for six months when a standard maintenance fix could have removed it, but I’m in a glass half full mood so I’m going look at it as a problem solved.

When I started this BLOG I wanted it to be a combination of SharePoint and none-work content. I assumed that SharePoint would take up the bulk of the posts as my personal life isn’t interesting enough to generate many readers, but up to now I’ve not made a single none-SharePoint post. That’s all about to change.

I’ve always fancied learning the guitar and so when my dad mentioned that he didn’t know what to buy me for Christmas I suggested one of these (another link

.) So for the last 4 weeks I’ve been spending half an hour every other night learning C, D and E chords. Tonight, like a true techie I threw out the instruction manual and decided I knew best. I found a site which claims to show you how to play Oasis’ Talk Tonight and set about teaching myself. 3 hours later my left hand is cramping, my index fingers is sore as hell and I’ve got this embarrassing video to share. I hope you all get a chuckle so that in 6 months time when I’m rocking with the Windows Vista band I can come back and say “I told you I’d make it.” If I can’t have that, I’ll settle for not being the next Numa Numa kid.

Enjoy

When I have a little spare time I like to look at my twitter feed for interesting BLOG posts. Often, if I find a well written article I’ll go back to see what the author has posted in the past. It was this process that caused me to stumble over this year old post by Mark Wagner (How To: Hide/Remove the View All Site Content link in SharePoint.) In the post he talks about using features to deploy custom style sheets, specifically to hide items on a page. While the process would certainly work,I think it’s a little “overkill” for a lot of scenarios, particularly in self service deployments where you want to avoid involving IT where possible. Here’s an alternative approach.

Let’s stick with Mark’s original idea of hiding the “View All Content” link. Create a test.css file and paste the following into it:

#ctl00_PlaceHolderLeftNavBar_idNavLinkViewAll      {display: none}

Save the file and upload it to the style sheet library on the target site, eg http://sharepoint-test/style%20library/. Now visit the site’s admin page and click “Master Page” Under “Look and Feel”.

Scroll to the bottom of the page and look for the section which allows you to select a custom style sheet. Click browse, navigate to the style library and select your test.css file. Once you click ok and leave the admin pages, you should see that the “View All Site Content” has gone.

You’ll notice that in this screenshot I’ve also hidden the recycle bin and the MySite link. Here’s the code you’ll need to replicate this:

#ctl00_PlaceHolderGlobalNavigation_ctl08_hlMySite  {display: none}

#ctl00_PlaceHolderGlobalNavigation_ctl06_hlMySite  {display: none}

#ctl00_PlaceHolderLeftNavBar_idNavLinkViewAll       {display: none}

#ctl00_PlaceHolderLeftNavBar_idNavLinkRecycleBin  {display: none}

Obviosuly this is useful for those of you interested in “look and feel,” but if you’re a support guy, it’s worth bearing this in mind if you get a support ticket for a site that’s “missing parts” or simply acting strange.

MCS UK have launched a blog

http://blogs.msdn.com/uksharepoint/default.aspx

I’ll be keeping my eye out for the guys I know. No slacking now fellas!

I’m interested in people’s opinions about automating Sharepoint installations. We aim to automate everything, in fact if someone needs to touch a key or mouse button we have to through a policy waiver process.

In principal I think this is a great idea. It allows us to have a “self service” server provisioning system where anyone can use a web site wizard and end up with a server with WSS all set up without too much understanding of the manual process. This allows us to devolve some disaster recovery situations out as all the engineer needs to do is use the self service wizard to bring a new front end server into the farm.

All good so far but getting to this point has been a painful process and involved some lateral thinking. I’ll try to explain some of the pitfalls we’ve hit and what we did to get around it. I’ll assume you’ve already automated everything up to the WSS/MOSS install, ie Windows, ISS, .NET etc and that you have a configured SQL server.

Application install

On the face of it this is the easiest step. Simply use the provided silent install .xml files and call setup.exe. The only gotcha is that you might want to consider slipsteaming your SharePoint hotfixes. The next step will be to either create a new farm or to connect to an existing configuration database. If it’s the latter, your servers need to be at the same patch level as the database so our automated hotfix scripts will copy the hotfix on to a share. The SharePoint install script checks to see if there are any files in that share and copies them into the slipstream directory before calling setup.

Barebones script

XCOPY \\share\hotfixes %SP_InstallScriptDir%\Updates

setup.exe /config silent.xml

Farm setup

Because we use the same process to perform the initial server build, planned rebuilds and DRing the server we need to first check if the farm has already been created. If you don’t do this and you come to rebuild a server you’ll get unexpected results. Most likely it’ll error and return a -1 code but it might set a property back to it’s day one value or in some rare instances it can nuke your config. Because we have to script using DOS batch, all I have access to is psconfig and stsadm. This means the only mechanism for checking if the farms exists is to run

psconfig -cmd configdb -connect

or

psconfig -cmd configdb -create

Back in the early beta days MOSS would flatten your config database if you ran -create against an existing farm. Once bitten, twice shy so I always use -connect. Obviously if the server is being rebuilt and the configuration database is intact then your server just rejoined the farm and the SharePoint is in the process of bringing it up to speed. If the config database isn’t there psconfig will return a -1 error to the console, so your code needs to look something like this:

psconfig -cmd configdb -connect -server %SQLServer% -database %ConfigDB% -user %ServiceAcct% -password %ServicePass% -cmd helpcollections -installall -cmd secureresources -cmd services -install -cmd installfeatures -cmd adminvs -provision -port 12345 -windowsauthprovider onlyusentlm -cmd applicationcontent -install

if %errorlevel% == -1 goto create

if %errorlevel% NEQ 0 goto error

if %errorlevel% == 0 goto end

Bear in mind that the account you call psconfig with must have permissions within SQL. For security reasons, Windows native runas command doesn’t allow you to pipe the password so if you either need to run your whole script as the service account or use something like SANUR or CPAU.

The rest of the arguments simply mimic the steps that the gui installer takes. Obviously tweak them to suit your environment.

Ok, so what if this is the first install or a farm rebuild? Well the above logic just dropped you into a routine called :create. The first job will be to create the configuration database

psconfig -cmd configdb -create -server %SQLServer% -database %ConfigDB% -user %ServiceAcct% -password %ServicePass% -admincontentdatabase %AdminContentDB% -cmd helpcollections -installall -cmd secureresources -cmd services -install -cmd installfeatures -cmd adminvs -provision -port 12345 -windowsauthprovider onlyusentlm -cmd applicationcontent -install

There’s your core Sharepoint installation with central admin on http://server:12345 . Everything else is specific to your environment so I go into too much detail, however I will touch on a few important points. Let me start with the limitations of stsadm – the command you’ll use for 99% of your configuration. For me, it has two main problems:

• It only offers a subset of the functionality of central admin, eg you can assign a quota template as the default for a web application but there is no command to create that template in the first place
• I’ve never been able to get it to return an error code other than -1, regardless of what the fault was. This is a real problem for unattended installs where you might want to put some logic into the script

Happily, Sharepoint MVP Gary Lapointe has release something in the region of 100 stsadm extensions and so point one is solved. I strongly advise you to install it at the start of your :create routine. There was a tool on codeplex called stsadm+ which tried to tackle point 2, but at the time of writing this it isn’t available. This means that if you have any failures  you can’t apply any logic and have to fail the script. This can be very frustrating if command 29 out of 30 fails and (assuming you’re sticking to your automated process) you have no choice but to disconnect the server from the farm, delete the database(s) and start over.

Be sure to run your script on one server at a time to avoid conflicts in the database. Make liberal use of

stsadm -o execadmsvcjobs

to ensure each command has finished. Below is an overview of the logic for this script:

Rem Slipstream any hotfixes that have been applied to the farm

XCOPY \\share\hotfixes %SP_InstallScriptDir%\Updates

REM Call setup to install Sharepoint

setup.exe /config silent.xml

REM Try to connect to the farm

psconfig -cmd configdb -connect -server %SQLServer% -database %ConfigDB% -user %ServiceAcct% -password %ServicePass% -cmd helpcollections -installall -cmd secureresources -cmd services -install -cmd installfeatures -cmd adminvs -provision -port 12345 -windowsauthprovider onlyusentlm -cmd applicationcontent -install

REM Check for errors to see if we connected

if %errorlevel% == -1 goto create

if %errorlevel% == 0 goto end

if %errorlevel% NEQ 0 goto error

:create

REM Create the farm

psconfig -cmd configdb -create -server %SQLServer% -database %ConfigDB% -user %ServiceAcct% -password %ServicePass% -admincontentdatabase %AdminContentDB% -cmd helpcollections -installall -cmd secureresources -cmd services -install -cmd installfeatures -cmd adminvs -provision -port 12345 -windowsauthprovider onlyusentlm -cmd applicationcontent -install

REM Install Gary Lapointe’s stsadm extensions

stsadm -o addsolution -filename lapointe.wsp

stsadm -o execadmsvcjobs

stsadm -o deploysolution -name lapointe.wsp -local -allowgacdeployment

REM Do your farm specific configuration

:End

REM Your error logic

:End

REM WooHoo we made it

Hotfix

There’s lots of info out there describing the SharePoint hotfix process so I’ll keep this brief. We split our script into two parts. Part one runs the hotfix and is run on all the servers at the same time. Part two is run on the servers one at a time and calls the upgrade process and then copies the hotfix to the share that I mentioned right at the start of this post.

psconfig -cmd upgrade -inplace b2b -wait -force

if %errorlevel% == 0 xcopy %hotfix% \\share\hotfixes\%hotfix%

I’ve not had the opportunity to convert all the above into a powershell script. It should solve the error logic gripe and would allow me much more control over the environment, effectively exposing what Gary Lapointe has given us via his extensions.

So back to my original point. Is all the above really worth it? Do you script your installs or do you do it manually? If you do use automation, what process are you using?

edit: Right on queue, SharePointdevwiki have just posted an article about scripting Sharepoint. Have a look