It’s been some time since my last blog post. This had mostly to do with some personal changes in my life and not really having interesting topics. But since that last post I’ve seen some “error in thinking” that most SP developers did, that I’ve decided to write a blog series about writing code and more importantly, writing code that does the job quicker or better (improving code quality).

Keep in mind that I never criticize the code of other developers, you never know in what mindset the code has been written. Were they on time pressure, lack of knowledge or is it more fundamental and they always write code the same way. Nothing is wrong with that, but as a consultant it is a good nature to question sometimes your own code and asking yourself, can’t this be done faster or is this the correct way?  

Continue reading

 

Well since SharePoint is such a big platform It is only natural that we display some curiosity to the other. Going from development to infra seems a big step and most people are reluctant to even look at the infra side of SharePoint. Most developers I know say that they don”t want and need to look at this side of SharePoint. Well here I believe they are wrong, as a developer it is more and more important that we know about the infra side of things. Not that we need to fully understand this side, but at least we have to  know the basics. With the app model It is becoming more clear that only coding isn”t sufficient to understand the platform.

Since I am doing more and more infra here a few things I have seen a lot and almost nobody that fixes them, while it is such an easy solution.

First Issue: Central admin is only accessible from the APP server and not from the WFE.

I see at many clients the same issue coming back every time. The administrator can access the central admin from the app server, but not from any other server. A credentials window comes up 3 times and doesn”t show anything.

But before I just tell you the solution.. Let”s do some investigation first..

Fiddler is your best friend in this.. I needed to know what happened with those authentication windows and why no page was shown.

So fired up my servers and using fiddler and IE to navigate to my CA address and see what”s happening.

Servers

IENotWorking

At this point it was clear that this wouldn”t work what so ever, going to test with another browser (Firefox in this case). And surprise , Firefox came with a credential question, filled in all the data and CA was loaded. Strange

FirefoxWorks

FirefoxWorksFiddler

As you can see, the authorization headers and authentication headers are different with IE and Firefox. The first one goes for ‘Negotiated’ and Firefox uses ‘NTML’.

Ok going to check the authentication provider on the CA web application…

CAAuthProvSettings

CAAuthProvSettings2

And just as I expected it was set to negotiate. Changed this to NTML and low and behold I could log in via the Web front end server to the CA.

ItWorks

Hope it helps for someone and that they don”t keep going to the App server just to be able to log on to the CA.

Edit 09/04/2014

Some time ago a buddy of mine (Koen Vosters, MSC) read this blog and noted a small issue with it.

1. The strange part is that it supposed to be negotiated, so meaning if Kerberos is not working, IE should fall back to the NTML way.

2. I should have defined SPN if I am using Kerberos.

A few days ago we had to export and import list items from a certain list, because we used item level security and when your items go beyond 5k item count than you will hit the threshold. Most of the advice that you find on the internet is “increase the threshold”.

Yes you can do that, but ask yourself, why is it 5000? Well this has something to do with the backend of the SharePoint system (which is SQL of course). SQL handles row locks up to a certain amount. More than 5000 and it does a lock on the entire SQL table, which is not good…

So back to our story, we redesigned the structure and used folders instead and see where the security was common. Ok, threshold limit fixed but we still needed to move the items to the folder.

Because we didn’t want to just copy the items and deleting the source item, we used SPExport and SPImport instead. Benefit here is that the entire item is exported, deleted and imported again on the correct location. Very important was the ID, this was a production environment and the mails that already were sent and maybe some lookups pointing to the list. So keeping the ID the same was very important.

Now it didn’t take long before we reached our first error.

Capture

After a quick search on the internet we find that when you have custom user fields defined via the elements.xml file like so:

   1: <;Field 

   2: ID='{A77F7435-0F70-44CD-9D4A-C10520E2E0B2}' Description='' 

   3: Name='LoginName' DisplayName='LoginName' StaticName='LoginName' Type='User' 

   4: Group='Meligo'/>

Now the code above will work and all is good in SharePoint land, until you are going to export / import an item.

I wanted to know from where the error comes and after some searching it in a certain area of SharePoint called with reflector we find this:

reflector1

reflector2

 

As you can see a check is being done if the field contains a lookup list and if that lookup list is equal to “UserInfo”. Great this is step 1 in solving the problem.

So some key attributes are missing from the schema. To fix the error above, you need to add in the schema of the field: ‘List=”UserInfo” ‘ .

Next, the export without compression gives you the directory below

Directory 

 

In this Directory the “UserGroup.xml” is the one that we should look at. this should contain as many user nodes as there are user fields:

xml

As you can see the “<User” tag is created for each user field in the item, when it is filled in. If it is not filled in, than that userfield is being skipped.

Now we should have 3 user fields here. At the moment this is not the case.

Still, this wasn’t enough. We’ve added the ‘List=”UserInfo”’ to the field schema and pushed the update to all the lists fields. But the user field was still looked at as a Lookup field instead of a user field.

It’s because the field is listed in the manifest.xml but there is no user field correctly parsed to it.

Well it turned out that the ‘mult=”false/true” ‘ needed to be added as well. Only then the field is being considered as an user field.

Good luck hunting for that one Smile

After we did the 2 changes in the field schema the import worked like a charm again.

Also if you get the error / warning “cannot find user with {ID}” than your user isn’t listed in the UserInfo list.

The User Information List can be accessed (Only if you’re admin) via the browser by navigating to /_catalogs/users/simple.aspx from your site. (Ex: http://YourSiteUrl/_catalogs/users/simple.aspx)

At the moment I’m sitting in the Gman garage in Antwerpen waiting on my car to be fixed. So excellent time to write some blog posts that I’ve been putting off for too long now.

Some time ago there was a problem at the client with a list of 10-20k items big. The discussion on how large a list is and should it be put in SP I’ll leave for another time, so the list was there and we had to solve it.

A simple query in an xsltlistviewwebpart would be the beginning of a big issue.

I’ve added some very interesting reads at the bottom of this article but all didn’t solve my issue.

Below was one of the first issues that we had.

Error_ItemLevelPermission

If you Google this one, a lot will tell, just set the threshold of the web application higher.

well, first ask yourself, why is this limit 5000 items?

To minimize database contention, SQL Server uses row-level locking as a strategy to ensure accurate updates without adversely affecting users who are accessing other rows. However, if a read or write database operation, such as a query, causes more than 5,000 rows to be locked at the same time, then it is more efficient for SQL Server to escalate the lock to the entire table until the database operation is completed. When this lock escalation occurs, it prevents other users from accessing the table

This small part in a very large blog article on Technet explains why.

Meanwhile the threshold was already set at 55000 items on production = NOT GOOD

Sure it solved the issue for a while but it created others elsewhere.

So back to the drawing board, why do we have this issue. Why is a query generating this particular error.

I will spare you the details on looking into this issue and skip to the solution.

It was because not only these 10k-20k large list was big but also all the items in it had broken security. So we had item level security in place. This is why we had this issue.

A little re engineering later (using folders with broken security and putting the items in those folders solved it) and all was good to go. From 20k items we went to 35 folders with broken security and the query would run nicely now.

A fun fact: working with large lists it’s often suggested to use the content iterator. Well the content iterator in the back-end applies a row limit, recursive node and listitemcollectionposition. In doing so it queries in batches but uses SPQuery just as well.

So CAML isn’t dead yet hehehe

 

Sneak preview: Did you know that our Caml Designer 2013 now supports SP 2010 as well, so you can work with the newly (and cool) designed CamlDesigner 2013 on both SharePoint versions. (Release is planned for 27 April).

 

Some good reading:

I’ve divided this into a table because of the many SharePoint versions:

Title (link to articles)

SharePoint Version

Comments

Software boundaries for large lists in SharePoint
2013

2013

 

Manage lists and libraries with many items

2013 & Office 365

 

Must do course

2013

 

Technet article

2010

 

Blog article from Steve Peschka

2010

 

Blog article from joel oleson

2010

 

Blog article from the SharePointBaker
(a buddy of mine 😉 )

2010

 

Working with large lists

WSS 3.0

It says WSS 3.0 but it still applies for the other
versions

Official read

2007

 

 

The location for the Codeplex project is here.

 

After messing around a lot with the elements files to generate some pages I’ve created a small webpart that does all this stuff for you.

I’ve reused some code from this codeplex project and I’ve asked Stefan Stanev if I could re-use it. Also I’ve added some functionality.

The codeplex project of Stefan connects to SharePoint and if you go to the page via de program than you can right click and generate the elements.xml file.

Only the program doesn’t take into account connections between webparts. This and some other small adjustment I’ve added in my solution. So the elements file that you can generate via my webpart should be the most complete one representing the page.

I’ve shared the code with Stefan so that he can update his package, but I don’t know if he did this or not.

So the install is easy, it’s a WSP so install, deploy and put it on an webpart page (example called: admin.aspx).

How does it work? Continue reading

Chalkboard-610

hey everyone

A couple of months to almost a year ago I had to write some documentation and lookup a lot around sandbox solutions. More precise when you would go for sandbox and when to farm solutions.

So I’ve restyled the document a little bit and you can download it as of now…

If there are some things you don’t agree or is incorrect let me know so I can change it…

Since I’m not all for re-inventing the wheel most of the information is a collection of blogs that I’ve found, all the links are saved in references at the bottom of the document.

Download:

Sandbox isn’t changed much in SharePoint 2013 so the document can be used for SharePoint 2010 and SharePoint 2013.

hope it helps someone

 

First of all, I would like to thank Walid Hadjadj to create this part, it was fun working with him on this project.

Ok, so we now know what to look for, soap calls when the InfoPath form is being saved back to the list.

If you want to read up on the formservice.asmx and the entire design, you can download the pdf file here. It’s about the only “complete” information that I could find on the net.

The soap call that we are interested in the most is “SetSchemaChangesForList”. This is the one that is responsible for telling the list to start using the InfoPath aspx pages that we’ve found in the previous blog post.

SetSchemaChangesForList

So first things first:

  • I’m re-using my previous solution, so if you don’t have it yet you can get up to speed by just downloading it here.

The extended lookup field, article unit price gave me issues with the redeployment of the InfoPath form, so I’ve removed it.

Also make sure that you download cablib, the file is needed to “extract” and “compress” the InfoPath file on a local storage drive (at the moment it is set to the C:windowsTemp folder, with cleanup functionality)

  • Make a reference to the web service “_vti_bin/formsservices.asmx”
  • Ok, a forms services helper is included so that the forms are being updated, checked and pushed to the lists (the formshelper.cs will be explained later in the blog)

At the moment all of the information is being pushed via the properties of a feature. Necessary data contains

  1. Location of the xsn file
  2. Content Type ID
  3. List url

only question remains: how are we going to get the InfoPath files in the new site collection …..

The only proper way to do this, is to provision the forms in a solution and deploy them to a document library, from there you can pick them up in code and deploy them to the lists

  • So add your InfoPath forms that you’ve downloaded to a directory in the solution and set the type to elements file. Also don’t forget to add the file to the elements.xml, so that the forms are being provisioned to a document library.

either via my admin webpart, where  you can just click “download InfoPath forms” or drag them in SharePoint Designer to a document library and download them from there

  • Your solution should look like the picture below solution

Ok , so the basics are there, now explaining how we did it.

First of all, there is a lot of logging to the SharePoint logs that I’ve implemented. So if you want to track what all the features are doing, you can use the ULS viewer and see all the lines being added.

  • Helpers –> formservices.cs

FieldSchema

This is just a default schema that will be used, if you look at the top print screen you can see the newfields tag and the update tag, so this is just a template that is being used later.

clientwebService

We need to instantiate a new forms web service object and bind the object to the current url the forms services asmx file.

The formsServices cannot work cross web, so if the list is listed in the web, you need to reference that web and put the asmx behind the web url.

Next is something very interesting:

designchecker

Design check form template is a WSDL Operation that checks if your InfoPath form is good or not.

The line above calls a method to get the form template. I’ll come back to this later.

Now for the next part you need CabLib.DLL, it is included in the zip package. But you can download it here or here if you want.

GetUpdatedFormTemplate1

  • First we need to retrieve the xsn file from the document library, I’ve passed the url value of the xsn file in the document via the fileUrl attribute.

GetUpdatedFormTemplate2

  • Since the InfoPath xsn file is in essence a cabinet file, it can be extracted and compressed by cablib.dll. But we need a location for that, so this is something to watch out for, I’ve chosen the temporary directory in the windows directory on the C drive. In that directory I’m creating a new one that with a random guid number as name. But first I’m checking if that directory already exists or not. This entire method is being run for each InfoPath file so a new directory is created each time with a new guid number of course. And at the end of the method a cleanup is done = deleting the directory.

GetUpdatedFormTemplate3

  • Next I’m using extract to extract the xsn file to the temporary folder. Also I’m loading the manifest.xsf in the Xmldocument object to manipulate the xml file.

manifest

  • Pay very close attention to the “PreserveWhiteSpace = true” line. This is very important. Apparently InfoPath cannot handle the following situation.

<Node>

         <SubNode> Value

         </SubNode>

</Node

  • InfoPath will give an error if you want to design it afterwards, because (and I’m not lying) the closing tag of “SubNode” is below the opening tag. The opening/closing of the “Node” tag isn’t a problem, but the “SubNode” tag is. This has taken up a lot of time to find and after a small pair programming session with Milanco (TL at the client) we found the issue. The location of placing the “PreserveWhiteSpace = true” is also from great importance. But it a line lower and it doesn’t do the same thing. You can check here why not Glimlach .

GetUpdatedFormTemplate4

  • Next up I’m going to change the first value and it’s called “baseUrl”, indicating the location of the xsn file in SharePoint structure.

GetUpdatedFormTemplate5

  • After setting the value of the “baseUrl” we are going to loop through the nodes looking for a node with the attribute “relativeListUrl”

GetUpdatedFormTemplate6

  • when we found the “relativeListUrl” we are getting the SPList object because we need to replace the hardcoded List ID with the new one

GetUpdatedFormTemplate7

  • we are also changing the contentTypeID, but I’m not sure if this step is required, because the Content Type ID will remain the same when deploying the CT via the same feature as you deployed to development environment
  • After all this the only thing that is left to do is to compress the directory back to an xsn file and return a Base64String back to the first method so that it can run through the Design Checker information object. And cleaning up the 2 directories as well.

GetUpdatedFormTemplateTheEnd

 

  • And last, we are pushing the initial value of InfoPath form location, ContentTypeID and relative list url via a feature activation with properties in the feature file

featureProperties

  • you call them via the feature receiver and just pass them to the formsservices helper

featureProperties2

This may look like an easy task but all of this needed to be investigated and took us almost 2-3 weeks to build. So I hope I’m saving some of you a lot of effort.

You can download the solution .

Want to learn more?

So, the foundation is there, we have our fields/ content types and lists. Now it’s time to create and make the InfoPath forms deployable.

Let’s go:

First, we need to “design” the forms… by default the new form of the item looks like the picture below, so a normal SharePoint new form.

OldNewForm

 

If you are getting an error in rendering the InfoPath form “The form cannot be rendered. This may be due to a misconfiguration of the State Service.

Than follow the instructions on this blog.

After some fun with the InfoPath designer, the form can look like this. Much better no?

NewClientform

 

So what does SharePoint do in the back-end to make this work?

  • it first creates (or uses) a folder on the lists directory, so you have “lists / list name / content type name / InfoPath pages”

OldAspx

 

  • In this directory (content type name) you can find 3 .aspx files and an InfoPath .xsn file

Newaspx

And these are the files that make up your InfoPath in browser form.

If you are now thinking that deployment is easy and you just need to create these files in the correct location and stuff… well think again..

Even if you put these files in the correct location AND you manage to change the hardcoded list ID’s in the InfoPath form than the SharePoint list will simple ignore this. The list needs to be told to use the InfoPath forms from now on, otherwise it will not use it.

Second problem is, how are you going to get the template.xsn? You can get it by going directly to the link and downloading the file. But are you going to do this for 80 lists? I guess the answer is no (or maybe)…

So I’ve created a small “admin” webpart to make your life easier. It checks if the file exists. If so than the template.xsn is downloaded to a document library (if the library does not exist than it’s being created) and renamed to the list-CTname.xsn .

So how does it all work?

When you click on the “customize form” button to generate the InfoPath form than a lot of stuff is happening in the back-end.

  • InfoPath is opened
  • some soap calls are being done to the lists.asmx and formservices.asmx

If we look at what happens via fiddler than you can see there are calls that are being done to the web services.

NewIPformWebServices

But these calls are when the InfoPath is being opened, we need to look at the calls that are being done when we are saving the InfoPath form to the list.

I’ve released a tool that generates the elements.xml file for you, just create a webpart page and use the tool to extract the module from it… you can find the blog post about the tool here.

So yesterday I had a whole day of fun with the xsltListViewWebpart = XLVWP, this is for SharePoint 2010 in SharePoint 2007 it’s called the listviewwebpart.

Situation:

Deployment of the module files that contain all kinds of webparts and always 1 or more XLVWP(s) .

But since we’ve added folders to the lists than this isn’t sufficient and we need to add “scope=recursive” to it as well.

Problems:

  1. XLVWP are not actually “deployable” (hardcoded list ID’s)
  2. The default view is always used, so even if you set it correctly, the architecture of the XLVWP will work against you

Solution:

This really depends on what you want to do…

Deployment of an XsltListViewWebpart

 

First things first, check out this post if you don’t know how to provision a webpart to a page.

Ok, now a first issue.. if you want to export a XsltListViewWebpart, you will have to do it via SharePoint Designer, it is not possible to export it via the GUI.

XLV-Web-Part-No-Export-Option

So fire up the old SharePoint Designer, go to the page, and select save web part.

SPDExportWP

Now you should have a .webpart file that should contain something like this:

FullElementsFile

After reviewing the entire xml structure you will see 3 things:

  • in the listName it gives a guid (list id)
  • in the webID property a null guid is given
  • in the xmldefinition property the url attribute is being filled in with the server relative url of the list.

So this is not good, the entire xml is too much and uses a listID? Yup, it actually expects to be redeployed with a single change.. Delete the list and recreate the exact same one.. the list id will be changed.. it will not work and give you an error message.

errorPage

Well happy hunting to what this could be.

you could add ?contents=1 behind the .aspx in the url, than you can see that one webpart is giving issues. 

ErrorPageContents1

Now the solution:

There are actually 2 possible solutions:

  • Either create a view tag and put in a minimal xsltlistviewwebpart like the picture below

viewtag

 

the view tag is a special one

On one msdn page you can see “The View element specifies list view Web Parts to use on site pages. “. So the purpose is to view lists.

but if you click on the view tag hyperlink you can read “Describes a view within a module for a site definition.”.

mmmm, so no listviewwebpart stuff? Glimlach

But you can define some other properties as well, like scope, rowlimit,…

  • the second solution to minimize the properties being given with the xsltlistviewwebpart

miniWP

Note the property name, ListUrl…

do not put an “/” in front of Lists, this will create an issue with the webpart that it cannot find the list.

Be careful when deploying the page more than 1 time, the webparts are just being added without rendering a new webpart ID. This will result in an error message saying that the webpart with the ID already exists. Best practice is to completely remove the pages and recreate them again via the elements file. Of course all changes made to the page in the meanwhile will be lost than.

Download the demo solution: icon_home_skydrive

I’ve created a codeplex solution for the generation of the webpart pages into a module (elements.xml) file. you can find it here

Some resources:

And a teaser for one of the next blog posts… Glimlach

In the last resource link a sentence stands out in the blog:

The short story of a lot of tries is: it does not work. Yes you can change the XmlDefinition property, but the XsltListViewWebPart does not use it after you have changed it.