Category Archives: SharePoint

All Developers Can Be SharePoint Developers

That’s the title for a full-day session I’m doing at the upcoming Philly Code Camp (October 10, 2015, Philly MTC in Malvern – more information).  Registration is only $76 and includes breakfast and lunch, raffles, guaranteed admission to the Saturday regular Code Camp (which will sell out), and more.  Register here.

Here’s the synopsis for the day:

Until recently, SharePoint was the red-headed stepchild of the Microsoft web development world, requiring intimate knowledge of SharePoint and it’s gargantuan object model. This made SharePoint a very specialized skill-set that stood off to the side on its own. It’s lack of support for standard web-development practices made it difficult for new developers to learn – or to even want to learn.

Fortunately, this has changed.

SharePoint 2013 has joined the rest of the web-development world – supporting standard technologies such as REST, and languages ranging from .NET to Perl, PHP, Ruby, etc. JavaScript is now a first-class citizen. Gone are the days when “doing SharePoint” required extensive, arcane knowledge. For developers, SharePoint is now just another data store accessible via a robust, REST-based API.

In this full-day session, SharePoint MVP David Mann will introduce the SharePoint REST API and show how it can be used to build client-side applications using JavaScript, utilizing popular frameworks such as Angular and libraries such as JQuery, common tools like Node, Bower, Gulp and Yeoman, as well as rich-client applications using C#. This includes both on-premises installations of SharePoint as well as cloud-based environments such as Office 365.

Bring your laptops as this is a hands-on session. No SharePoint experience is necessary, but basic programming skills would be a big help. This is a great opportunity to add SharePoint to your toolkit, or to take your SharePoint skills to the next level. Individuals and teams are welcome.


I’m still tweaking the material, but here is the current agenda:

  1. Introduction & Getting Started (including why SharePoint? )
  2. Setting up the Environment
  3. Setting up the Environment (Lab)
  4. Tools & Frameworks
  5. Tools & Frameworks (Lab)
  6. The SharePoint REST API
  7. Hello (SharePoint REST) World (Lab)
  8. CRUD Operations
  9. CRUD Operations (Lab)
  10. Provisioning
  11. Provisioning (Lab)
  12. Language Agnostic – C#, Ruby, [Insert favorite language here]
  13. Language Agnostic (Lab)
  14. Wrap Up and Next Steps

I’ll post more details as we get closer.

To follow along with the labs, you’ll need:

  • a laptop with Visual Studio (2013 or 2015)  installed

That’s it.  You will not need SharePoint installed

Critical Path Training Webinar Material

I completed my Critical Path Training webinar earlier today.  All in all, I think it went pretty well.  There was a TON of material to cover, and as usual, not enough time to cover it all.  I’ve posted the source code I showed to Git (https://github.com/Sector43/CPTJSWebinar) and you can clone it from here:  https://github.com/Sector43/CPTJSWebinar.git.

In the webinar I showed a sample application which manipulated Views in SharePoint to show an icon in place of a column value based on the value of that column for the item in the view.  It wasn’t a fancy demo, but the code has a lot of good stuff in it – including a full suite of tests, use of the revealing module pattern, IIFEs, defensive coding, dependency injection, etc. 

Once you’ve pulled the code down, there are a few steps you’ll need to take to make everything work.  As I said in the demo, this is not beginner-level stuff, so I’m assuming you’re familiar with VS, node, NuGet, etc. or can figure out what you need.  Here’s a summary of what you need to do

  1. Install node – www.nodejs.org
  2. Add an environment variable for CHROME_BIN pointing to the folder where you have Chrome installed.  You can also add one for FIREFOX_BIN if you’d like to use Firefox for testing in Karma.
  3. Add the Karma Visual Studio plugin from here: https://visualstudiogallery.msdn.microsoft.com/bfe6feb7-7ec4-4e8e-9d90-cf6ea2cd2169  (only if you want to see the test results in the VS Test Runner window).
  4. In Visual Studio, toggle the “Run Tests on Build” option on in the Test Explorer Window
  5. Restore NuGet packages
  6. Install node packages (from a command prompt in the root of your project):
    • npm install karma-cli
    • npm-install karma
    • npm install karma-chrome-launcher
    • npm install karma-xml-reporter
    • npm-install karma-jasmine
    • npm-install jasmine-jquery
  7. Edit the karma.conf.js that is in the source code if necessary
  8. From the command prompt in the root of your project: karma start

Karma should now launch in a command window, launch Chrome and then run all of  the tests in the project.  They should all pass initially.

Feel free to kick the tires on the code and give it all a good workout.  If you find bugs, please issue a pull request and I’ll work them back into the repo.

The LogManager and REST utilities are included in here but will eventually be wrapped into the SPClientLib project I’m working on (which will also be released on GitHub).

As I said in the webinar, this is what I would consider “production-ready” code, but PLEASE review it and understand what it is doing before using any piece of it as-is.  It is released under the MIT license so you are free to use it for any purpose, but you must keep attributions intact and please share back any improvements.

Dave

Part 2: Loading JavaScript Libraries (It Still Shouldn’t Be This Hard)

My post on Sunday seems to have struck a little bit of a nerve.  A number of folks added comments to the blog post and a few others commented directly to me via email or other channels.  In this post, I’m going to quickly summarize and respond to some of the comments and then move on to step two – trying to formulate thoughts for moving forward, if there is in fact a path to move forward any further.

OK, here we go, in no particular order…

(Oh, yeah, perhaps this goes without saying, but these are largely opinions based on my experience and stories I hear from other developers.  YMMV, and I’d *love* to be proven wrong about some of these, but let’s make sure to keep the discussion productive and moving towards a solution).

  • Marc’s concept of “Functions as a Service” (FAAS) is interesting and one I think should be investigated and fleshed out.  I haven’t read the book chapter Marc references, so maybe he’s already done that…Marc?
  • Hugh’s mention of JSFiddle as a case study is *exactly* what I had in mind for a part of this.  For small-scale customizations implemented by an end user, selecting a library or two needs to be as dirt simple as JSFiddle.  I think this ties in to FAAS as well.
  • As a few folks mentioned, this is not entirely a technical problem.  Regular old Governance comes into play in a big way.  Too, if anything comes of this, we must get a groundswell of community support behind it as a “standard”.
  • This is perhaps a semantic argument, but contrary to what a few folks posted, I think there *is* a one-size-fits-all solution, or at least one-size-fits-most – there’s always going to be edge cases that don’t work.  The semantics comes in because I’ll call it one solution with a couple of facets, but all falling under one umbrella.
  • SPO (Office 365) and MDS must be addressed
  • Supporting MDS is somewhat difficult, and more than just not polluting the global namespace
  • RequireJS is a great example of managing dependencies and loading modules. I use it frequently, as do many developers. Unfortunately, as it stands right now, it is not MDS-compliant (it creates script nodes using CreateElement)
  • Loading multiple versions of the same library is a performance and memory use problem, but also breaks some libraries (like JQueryUI, and probably a bunch of other JQuery plugins, though I haven’t tested them).  Namespacing doesn’t fix all of this.
  • The other problem with loading multiple libraries is also compounded by a lack of control over load order.  SharePoint has too many ways to load libraries and they all load at different times.
  • A comment was made that “most companies would not want to advertise the existence of a “standard library loader” to general users… since that would implicitly suggest that ad-hoc customization is ok…maybe even supported” which part of me agrees with, but a bigger part of me thinks those companies are just burying their heads in the sand.  Unless they somehow hide the Script Editor webpart (which some do, I know) they have people doing customizations and ignoring it doesn’t make it go away.  They’d be better off embracing and providing some guidance/support.  This also plays into my “one solution with many facets” approach – companies can choose which pieces they wish to implement to cover their scenarios.
  • Another comment was “As a developer you should be working to make YOUR js libraries isolated from anyone else.”  I think this is wrong.  This is not being a good corporate citizen.  This is the “too hell with everyone else” scenario in my original post.  I agree that you should isolate your code as much as possible, but unless everyone is there can still be problems.  Too, your code is not the only code running in the environment.  The only exception to this I can think of is if you are developing an application as an ISV or internal developer/consultant and your application is entirely self-contained – you control the horizontal and the vertical.  However, if you develop even one web part as part of your application, you don’t have that level of control – users could add that webpart outside of your application.  If you integrate with OOB SharePoint – even just using its Master Page – you don’t have that level of control.  If your application allows Custom Actions to load, you don’t have this level of control…etc., etc., etc.  Even if your stuff is perfectly insulated and always works, it may break something else or cause some other problem.
  • A few folks commented along the lines of user’s maintaining/understanding/impacting their code.  This falls under the previous bullet.  In most cases, your code is not the only custom code running in the environment
  • A big part of the problem is that we cannot say that this is only of concern to professional developers.  While I personally dislike the term “citizen developer” I agree with the fact that non-developer users are customizing SharePoint and using JavaScript they cobble together from the internet or simple trial and error, and they too often “ship it” as soon as it appears to do what they want with no additional testing or documentation.  This ship has sailed and there’s no way we’re bringing it back to port, much as we might want to.  (And yes, I realize that some of those “citizen developers” do very good work – better than some developers – but I’m mostly concerned with those who don’t)
  • I have not specifically noticed the “jarring effect” some folks mentioned, but maybe I’m just missing it.  I’ll dig out some code I wrote that should show it and try to pay more attention.

I think that sums up my responses and follow-on comments.  Now to move forward…

As I see things in my twisted little mind, here’s what we need, from a technology point of view (governance is next, don’t worry).  This is very much of a blue sky wish list and I realize that some of it may not be feasible/possible.

  • A replacement for the Script Editor web part that simplifies things for end users to allow them to pick the libraries they need and guides them towards some minimal “best practices” – a very lightweight linter of some sort that enforces a few basic rules would be a good start.  Optionally blocking inline script tags would also be nice (so MDS doesn’t break).  I think 99.9% of the time, simply telling them that JQuery is already available will be sufficient, so this doesn’t have to be fancy.
  • A way to track which libraries and versions are available globally (scoped to a given web, I think)
  • A way for developers to specify/discover which libraries are loaded and the order in which they load
  • A fast, reliable way to load and initiate a script loader which can then load other required scripts
  • An MDS-compliant way to specify dependencies between modules and load order

I’ve probably missed a few things, but that’s a start.

For governance, here’s my wish list:

  • Documentation of whatever the technology solutions are
  • Consensus amongst developers that whatever “this” is, it is the recommend way to proceed (although I acknowledge this is like herding cats and will never be close to 100%)
  • Training/education material for end users and developers

So where do we go from here?  Next is a viability assessment of the technology wish list.  What’s possible?  What’s feasible?  After that, if enough of the technology can be made to work then we kick off a community project.  Or we talk to Microsoft about carving off a little corner of the PnP world for this effort and get a little help from the mother ship (which would also go a long way in getting buy in from developers and companies).

This may all very well be a pipe dream, but I’m somewhat known for tilting at windmills (and metaphors, mixed or otherwise) but if we don’t at least try to come up with a solution to a problem, all we’re doing is complaining and that’s just wasted energy.

As always, comments welcome.

Dave

Loading JavaScript Libraries – It Shouldn’t Be This Hard

I’ve been doing a lot of JavaScript programming for the past few years. SharePoint 2013 has only accelerated the pace. Currently, I split my time roughly evenly between SharePoint and non-SharePoint work at my primary client, but both projects make heavy use of JavaScript, between them making use of many of the popular JavaScript libraries – Angular, JQuery, JQueryUI, Moment, Require, Knockout, etc., etc., etc.

The disparity between the two worlds is still quite wide. Even something as simple as loading up required libraries is more difficult in the SharePoint world than in the rest of the world. Part of this, I realize, is due to the nature of SharePoint. It is a much larger application than many other non-SharePoint applications and lends itself to extensibility much more than those non-SharePoint applications. There are often far more people involved as potential customizers of SharePoint than with non-SharePoint applications

And that’s where the trouble begins.

Extensibility means a loss of control. Even in controlled environments that actually effectively do SharePoint governance, extensibility invites problems. JavaScript is still a little bit of the Wild West; every project is going mandate the use of different libraries, different versions and different configurations. If all I’m concerned about is my project, things are easy: I load the libraries I need using one of any number of script loaders or script-loading approaches. To hell with anyone else.

That, obviously, is not a very good approach, but it is one taken all too often. I was recently asked to look into a problem at a client that boiled down to them having a ridiculous number of JavaScript libraries loaded on too many pages:

  • 4 different versions of JQuery
    • One page actually loaded JQuery 6 times, as poorly trained site admins, or users with permissions, were adding Script Editor Web parts on pages and each loading JQuery, from different document libraries and CDNs
  • 2 different versions of JQueryUI
  • 2 versions of Knockout

This is in an environment that at least makes an effort at governance. Unfortunately, too much of it is left in the hands of developers, but this isn’t a post about governance directly. Instead, it is an attempt to try and figure out what we as developers and architects should do.

As a consultant and ISV, I cannot mandate too strictly how my client or user’s environment be configured. I can say something like – if you don’t have JQuery installed and loaded, we will install and load it. But that’s about it. That may be fine for the moment my customization (either custom built as a consultant or a product I sell as an ISV) is added to the environment, but I have no control over what happens later. If, three days after my customization is added, another product or customization is added which loads its own version of JQuery we may have a problem – especially with the somewhat recent split of compatibility between JQuery 1.x and 2.x. The one that prompted me being called in to that client was caused by a customization loading a new version of JQueryUI, one which contained a different custom collection of widgets. It broke a different customization.

What’s the solution? I don’t have a good answer, but I’m working on it and invite collaboration. If we can come up with something good, this would be a killer community-driven project. Here are the problems, as I see them:

  1. We need some way to track which libraries and versions are available in an environment. This can’t be “just” documentation or offline means – code should be able to check and respond appropriately. Somehow, this needs to be global across the environment and future-proof.
  2. MDS must be supported. As a consultant or ISV, I cannot dictate that MDS be turned off and have that remain inviolate for all time. I know many developers prefer to just turn it off, but it is a feature of SharePoint and therefore must be supported.
  3. There must be a standard for how libraries are loaded so you can anticipate when they will be available. One of the problems faced by the client I mentioned was that via the convoluted mess they had gotten themselves into, JQueryUI was loading before JQuery, which doesn’t work.
  4. There must be a simple way for non-developers to participate in maintaining things. Otherwise we run into the problem above where users had loaded many different versions of JQuery via the Script Editor Web Part
  5. It would be nice if this made things easier for developers – perhaps by supporting AMD modules and requirements management, either via Require.JS directly or a fork of it to make it SharePoint and MDS compatible.
  6. It should support both on-premises and SharePoint Online/Office 365, including Apps.

There are a few other things that should probably be included in there, but these, to me, are the big targets. I realize that there are answers to each of these individually, but what is missing is a comprehensive approach and a commitment from the community to drive a consistent approach. Honestly, there may not be one ring to rule them all. The best thing may be two or three approaches, but that would still be better, especially if there were some consistency and commonality amongst them.

If anything is to come of this, education and community support are critical to making it work, so what do you think? Leave a reply and we’ll see where this goes.

Dave

SPC423 Recap: Deep Dive – REST and CSOM Comparison

I presented my session at the 2014 SharePoint Conference today (I’m typing this while sitting on the plane on my way home) and as promised, here’s the write up. This will include all of the material I presented at SPC as well as some additional material which I did not have time to include in my session (sessions were 75 minutes long, and I watched the timer tick down the last few seconds as I was wrapping up my final slides. Talk about perfect timing!). In addition to this write up, I’ll also provide links to my source code at the end of this post. I’m not providing the slide deck here as it would simply be extraneous – all of the material is covered in this post. Sit back and get comfy, this is going to be a long one.

Here’s what the session was all about:

SharePoint 2013 is fully embracing the world of client-side technologies, both on-premises and in the cloud. Regardless of which programming language you use (C#, VB.NET or JavaScript) a fundamental decision you must make is which API you will use to reach back into SharePoint – REST or CSOM? This session will briefly introduce each API, and then quickly move into a demo-heavy comparison of both approaches, citing the pros and cons of each. By the end of the session, you will have a solid understanding of when and why you would use one API over the other (regardless of programming language) as well as plenty of examples of each in action.

The main focus was to help developers pick an API with which to access SharePoint from off-server. There were quite a number of announcements in the conference and developer keynotes related to new capabilities which, when combined with existing confusion, inaccurate information and FUD, often led developers to make bad choices with regard to their API. Actually, scratch that. As I tried to get across in the session, it’s usually not a bad choice, it just may not be the best choice.

There are very few bad choices between the APIs – one API may just be a better choice in a particular situation.

Before going any further, I should at least mention the other popular option for SharePoint development. If you have the choice to run code on-server, then you’d be using the SSOM – Server Side Object Model. This is the one Microsoft is shunting to the side with their focus on “THE CLOUD.” This is the one with which most of us SharePoint developers are most comfortable. I won’t say that the SSOM is never going away because that is obviously not my call to make. However, so long as there is an on-premises version of SharePoint, there will be an SSOM. Truth be told, the SSOM is currently the only one Microsoft actually writes and maintains – the other two are generated from it. So you could think of the SSOM as Amber, and CSOM and REST mere Shadows. (An obscure reference for the geeks reading this, but essentially what I’m saying is that the SSOM is the source of both CSOM and REST).

Anyway, that’s not what I came to talk about. I came to talk about the draft REST vs CSOM (anybody catch that reference?). REST and CSOM are each perfectly valid choices for interacting with SharePoint from a remote location – essentially anywhere except the SharePoint server itself. With two “perfectly valid choices” how is a developer supposed to know which is the right one to use? Is there a right one? Why are there two to begin with?

That’s why this session exists. Let’s dig in and get started answering those questions and more…

First of all, there are two APIs because they have different approaches, different capabilities and different situations in which they provide value, and that’s the reason this session was created – help devs to understand the differences and know when to pick one over the other.

People will often ask why I use “CSOM” when referring to a JavaScript example or situation. My answer is that the acronym CSOM encompasses all three of the client-side object models – .NET, Silverlight and JavaScript. So when I’m talking about something that applies to all three, I use CSOM, even if my example is in JavaScript. Technically, the acronym JSOM, which many people use to refer just to the JavaScript implementation, is inaccurate as there really is no such thing as a JavaScript Object Model; there is just the client-side object model implemented in JavaScript. That’s really splitting hairs and I do use JSOM when I’m talking only about the JavaScript implementation of CSOM, or about things which only apply to the JavaScript implementation. Along the same lines, I’ll use .NET CSOM if I’m talking exclusively about the .NET implementation.So, to summarize:

  • CSOM = all three implementations, .NET, Silverlight and JavaScript.
  • JSOM = just the JavaScript implementation
  • .NET CSOM = just the .NET implementation

And no, there really isn’t a Silverlight-specific acronym as no one really talks about Silverlight anymore.

There are five basic elements that should be considered when you begin a project involving remotely accessing SharePoint with regard to selecting an API set. If I were to rank them in order of typical performance for the projects I work on, they would currently look like this:

  1. Technology Concerns
  2. Performance
  3. Testability
  4. Intellectual Property Protection
  5. Developer Concerns

For any given situation in which you are likely to find yourself, you may rank these differently, and that’s perfectly OK. I’d rank them differently in some situations, too, but most of the time I think they go in that order. We’ll touch upon each of these in the rest of this article. But first I want to deal with some that didn’t make the list:

· Security – someone shouted this out during the “game show” portion of the session (you had to be there) and honestly I was a bit flummoxed. My first reaction was “how did I miss something as important as security?” But when I had a moment to think about it, I realized that security didn’t make the list because there really is no difference between the API sets. SharePoint is always the final arbiter of security for content it holds and the API with which you access SharePoint doesn’t come into play here – each can work with SharePoint in a secure manner. A related concern can come into play but that’s a language choice, not an API choice. If you’re writing your REST or CSOM code in JavaScript, then your security options ARE limited. Essentially you can’t do impersonation in JavaScript as it always runs in the browser context. This, however, as I said, is a result of your language choice – JavaScript – not your API choice – REST or CSOM. In .NET, REST and CSOM each allows for impersonation.

· Browser Choice – this one didn’t come up in the session but I want to call it out here. Some people might think that the browser you choose is going to impact your CSOM or REST code and that is actually not true. For any browser which SharePoint supports (see here), both REST and CSOM will work perfectly well.

· State Management – I find this one interesting because in my experience few, if any, devs take advantage of this. Essentially, what it comes down to is that CSOM can maintain the state of your objects across calls back to the server. I cover this in my Pluralsight course and it is definitely a nice feature when you need it. As I said, however, most devs don’t know about it and it’s not really a very strong reason to pick one API over the other

OK, so let’s start looking at the element we do need to consider.

Technology Concerns

This concern breaks down into several sub-areas that are going to come into play in most situations. The one commonality across them is that they are typically not things which can be changed. They are environmental and typically much larger than simply accessing SharePoint for one project.

Low-Hanging Fruit

These are the easy ones because it’s completely cut and dry – things either work or they don’t. These fall completely on the pro-CSOM side as they are things that simply do not (yet) work here for REST. If you need to access either Managed Metadata or Workflow, you cannot do so with REST and so you must use CSOM.

Technology Stack

This element concerns your basic computer (server, workstation, tablet, etc) platform and which tools or libraries you may be using. For our purposes here, computer platform breaks down into:

1. Microsoft

2. Non-Microsoft

We can drill into this a little further to encompass more specific gradations such as iOS, Android, *nix, etc.

Tools and libraries generally break down to “those that were made for SharePoint” (specifically an implementation of the CSOM) and everything else. From the JavaScript side, this includes everything from jQuery to a utility library you write by yourself. For .NET, this ranges from something like Entity Framework to again, a simple library you write on your own.

While you can generally make either CSOM or REST work across any of these scenarios, in general, REST is going to work better for the non-Microsoft platforms and third party libraries. It is unlikely worth the effort to run Mono just to get access to the .NET CSOM and while downloading the JSOM wouldn’t be that difficult, it’s only going to work in browser applications. If you’re making significant use of any third party libraries, the effort required to make the JSOM work (things like collections and objects, etc.) is unlikely worth the benefit provided by the JSOM. Especially when you have a perfectly valid REST implementation available.

If you’re on a Microsoft platform and/or only using SharePoint-specific libraries, then either CSOM or REST are equally useful.

Role of SharePoint

This technology concern covers how you are using SharePoint:

· As simply another data-store – the vast majority of what you’re code is doing is simply reading and writing data that just happens to be stored in SharePoint.

· Application pieces of SharePoint – your code spends more of its time working with SharePoint-specific constructs – sites/webs, lists/libraries, security, users, etc.

This element is a little harder to call a winner for because both REST and CSOM can operate equally well here. However, if SharePoint is just another data-store for your application, it is likely to be a little easier to use REST as it works with more traditional data structures – XML and JSON. If you’re interacting more with the application elements then CSOM has a slight advantage due to its similarities with the SSOM; it will be marginally easier for developers to make the client-side transition from the SSOM since the objects are going to be so similar.

Deployment Mechanism

How and where you are going to deploy your code is another technology factor to consider and it’s another that’s pretty cut and dry:

· If you’re deploying full-trust solutions, then most of your code is probably SSOM, but for the client side elements, you could do CSOM or REST equally well

· If you’re deploying apps:

o SharePoint-hosted apps: these tend to be more data-centric (though certainly don’t have to be) so to a certain extent, we fall back on the “Role of SharePoint” element and so choose REST. CSOM would also be a perfectly valid choice, especially if your app interacts more with SharePoint as an application. You’re using the JavaScript implementation of one API or the other here.

o Cloud-hosted apps: This also falls back on the “role of SharePoint” but also gets impacted by the technology stack in use. If you’re on a Microsoft server, I think CSOM is a better choice. If you’re on a non-Microsoft stack, then REST in whatever language is supported by your server is probably best.

Technology Target

This last element here is pretty straightforward. There are four scenarios we’ll look at:

1. SharePoint, either on-premises or in the cloud

2. Non-SharePoint elements of Office 365 (Exchange, Lync), including the newly announced “Office 365 platform” API

3. Other Microsoft Cloud Applications – Dynamics, Project, etc

4. Non-Microsoft Cloud Applications – Salesforce, etc.

This one is pretty easy as CSOM supports only SharePoint, either on-premises or in the cloud. While the others may support some flavor of client side API, it is not technically the “CSOM” in the sense we’re talking about. This includes the “Office 365 platform” unveiled at SPC14 – that is available only via REST.

So – CSOM or REST are an option for SharePoint, for anything else, you’re using REST.

This is a good time to stress an important point. Microsoft is committed to CSOM for SharePoint. I confirmed this with a few folks at SPC. CSOM isn’t going anywhere anytime soon, so we’re safe using it now and for the foreseeable future.

Performance

Performance is important for any application and comes in second place here only because the elements of the first place Technology Concerns cannot often be changed. We must get the best performance we can within the confines of the technology concerns with which we are stuck.

For our purposes here, performance is also going to encompass user experience as it is nearly impossible to have a good user experience with poor performance.

Performance when you’re conducting round-trip calls from client to server and back again is mostly concerned with how often you hit the wire, how much data flows across the wire per call and the total time required to complete the calls. Those are the three factors we’re concerned about here. In one of them, CSOM wins hands down, in the other two REST wins.

CSOM wins its portion because it has to hit the wire far less often than REST. Compared to CSOM, REST is exceedingly chatty. In the tests I performed, REST made about 70x more calls back to the server than did CSOM. This is because currently, REST has to call back to the server for each and every operation whereas CSOM allows you to batch calls (within some limitations) and send a series of operations to the server in a single call. This makes CSOM much more efficient.

I tested a series of pretty common operations in both REST and CSOM:

  • Creating Lists (10)
  • Creating List Items (100 per List, for a total of 1000)
  • Reading List Items by ID
  • Querying for List Items (retrieving 50 per List)
  • Deleting List Items
  • Deleting Lists

To complete these operations, CSOM made 42 calls back to the server with a total of about 2.5MB traversing the wire and an average payload size for each call of a little over 40K. REST, on the other hand, required over 3000 calls with total network traffic of a little more than 5 MB, but had an average payload size of a little less than 2K. Overall, each REST call was far smaller than each CSOM call and about 85% of each REST call was comprised of the HTTP Headers. If we could batch our REST calls (which IS supported by the underlying architecture but not likely something we’ll see soon) then REST would be much more of a contender here. By batching our REST calls, we’d have less calls and therefore less header traffic.

The other interesting thing here is that despite requiring more calls and more traffic (again, mostly headers) REST outperformed CSOM from a total time perspective by about 30%.

In a nutshell, there are plusses and minuses for both REST and CSOM from a performance perspective. While in general, we would award the winner to whichever took the least time (REST), the sheer number of calls made by REST and the 2x amount of data flowing across the wire were too much to overcome and we have to award this round to CSOM. For your situation, the operations you perform may not require as many calls and so REST will be a more logical choice. Remember, on a per-call basis, REST outperformed CSOM for both time and payload size, so if you don’t require as many calls, REST can outperform CSOM.

Testability

Testability is important. As I had one my slides in the SPC talk, there’s a little developer haiku I wrote which reflects this:

Codebase fluctuates
Unit Tests protect and calm
This ship has sailed

Unfortunately, as SharePoint developers, we tend to be woefully behind the curve in unit testing our code. For that reason alone, this particular element is bumped up in importance to being a major factor to contend with when picking an API.

Historically, SSOM has been exceedingly difficult to unit test for many reasons (sealed classes, internal constructors, etc). That has led a lot (most?) SharePoint developers to not bother unit testing their SharePoint code.

With the move to the client side, we have the opportunity to remedy this. Hopefully, the SharePoint community will take this opportunity to start unit testing more of our code.

With that goal in mind, let’s examine the two APIs from a testability point of view. While I’ve tried to stick with facts as much as possible for this article so far, testability is an area that is really a matter of opinion. Largely this is so because either API is equally testable so it really comes down to which you find easier/better. At SPC, the afternoon before my session, Steve Walker (from the SharePoint product team) had a session in which he stated that unit testing CSOM code was easier than REST code. I find the exact opposite to be true. Steve didn’t go into details on his statement, but for me, I felt that I had less test code to write to work with REST and what I did have to write was easier and more “web development” standard than what I had to write for CSOM.

In either case, a good quality mocking product is required. In my session, I showed Telerik’s JustMock, my current favorite for mocking. In the past, I’ve used TypeMock’s Isolator product for SSOM work, and there are numerous other mocking products that ought to work well, too. The mocking tool is required because we need to be able to have our test code function without the dependency of a SharePoint server. We do this by creating fake objects (mocks) as stand-ins for the real objects. For these fake objects, we control the horizontal and the vertical; we control every aspect of their existence – from what their methods do to what their properties return.

For example, when mocking .NET CSOM, I had to set up fake objects for ClientContext, Web, ListCollection and List for my demo, like so:

var mockCtx = Mock.Create<ClientContext>(Constructor.Mocked);

var mockWeb = Mock.Create<Web>();

var mockLists = Mock.Create<ListCollection>();

var mockList = Mock.Create<List>();

Figure 1: Mocking .NET CSOM with Telerik’s JustMock

Then when instantiating or accessing things, I needed to make sure that my mock objects were returned instead of the real objects, or in some cases (the Load and ExecuteQuery calls) that the mocked objects did nothing:

Mock.Arrange( () => mockLists.GetEnumerator()).IgnoreInstance().Returns(getEnum());

Mock.Arrange( () => mockCtx.Web).IgnoreInstance().Returns(mockWeb);

Mock.Arrange( () => mockWeb.Lists).IgnoreInstance().Returns(mockLists);

Mock.Arrange( () => mockCtx.Load<ListCollection>(mockLists)).IgnoreInstance().DoNothing();

Mock.Arrange( () => mockCtx.ExecuteQuery()).IgnoreInstance().DoNothing();

Figure 2: Configuring operations and properties hen testing .NET CSOM with JustMock

With the help of one more utility function, my CSOM test was complete:

private IEnumerator<List> getEnum()

{

    for (int i = 0; i < 20; i++)

    {

        var mockList = Mock.Create<List>();

        Mock.Arrange( () => mockList.Title).Returns(expected.ElementAt(i));

        yield return mockList;

    }

}

Figure 3: One final important function to round out my .NET CSOM unit test

(Full source code for the tests, as well as all of my session demos, is available at the bottom of this article)

With all of this in place, I could test my .NET CSOM code as part of my Unit Tests without requiring that the SharePoint server be involved, or even running. In my SPC demo, I shut down the W3C service before running the tests to prove that the mocked objects were working properly.

Contrast the above code with what it required in REST (again using JustMock) and you’ll get an idea of why I feel testing REST to be a little easier:

var mockRequest = Mock.Create<HttpWebRequest>();

var mockResponse = Mock.Create<HttpWebResponse>();

Mock.Arrange(() => WebRequest.Create(Arg.AnyString)).Returns(mockRequest);

Mock.Arrange(() => mockRequest.GetResponse()).Returns(mockResponse);

Mock.Arrange(() => mockResponse.GetResponseStream()).Returns(GetFakeResponseStream());

Figure 4: Testing REST with Telerik’s JustMock

There is actually one supporting method for the REST test (GetFakeResponseStream) that is not shown here. It is pretty simple but I didn’t reproduce it here because it contains one very long string (the content of the response) that wouldn’t show very well here. You can see it in the full source code listing.

In general, I found testing REST to be easier because it required less lines of code and because I was really dealing with testing standard web objects that are going to be far more familiar to most devs. The SharePoint-specific objects in CSOM were just a little bit harder to work with. You may feel differently, and that’s OK.

The good news is that neither API is particularly difficult to test, so there is really no reason not to test your code. Please, please, please begin writing unit tests if you’re not already doing so, or write more if you’re already writing some. The overall quality of your code WILL go up, and that’s not a bad thing.

Intellectual Property Protection

If all you do is write code for use by your company, then this element is likely not very important for you. With the introduction of apps and the SharePoint app store, however, more and more developers are beginning to produce products for use in SharePoint by others outside their company. Suddenly protecting the fruits of your labors – your intellectual property – is now important to you. For this reason, combined with the fact that it is something many of us have not had to deal with much before, it is called out here.

Protecting your IP is all about keeping someone who may have a legitimate license to have your application (either because they paid you for it or because they’ve downloaded a trial version) from reverse engineering it and reusing the code to produce their own application which may or may not compete directly with yours.

In the SharePoint world, we’re primarily talking here about Apps. Perhaps these are apps they’re purchased or acquired through the SharePoint store, perhaps ones they’ve purchased from your website and installed by themselves. In a smaller number of cases, these may be full-trust solutions they’ve purchased from you and installed on their on-premises farm, but mostly we’re talking about apps here.

As you should already know, there are primarily two types of apps – SharePoint-hosted and Cloud-hosted (the latter encompassing both provider- and auto-hosted).

For SharePoint hosted apps, there is not any way to really protect your IP. Everything runs on the client (SharePoint-hosted apps cannot run server code) and so all of the HTML, CSS, and JavaScript is downloaded to the client. This makes it all available to someone to inspect and do with what they choose. True, there are ways to obfuscate some of this and make it more difficult, but for every obfuscator, there is a corresponding de-obfuscator. The best you can do here is make things a little more difficult and maybe not worth the effort for someone trying to reverse-engineer your application. If you’re lucky, maybe they’ll look at yours, decide it’s going to be too hard to steal, and just move on to another app.

In my session I talked about one somewhat interesting use of something like TypeScript would be to complicate your JavaScript just enough to make it too much work for someone trying to steal your IP. Typescript, if you’re not familiar with it, generates 100% valid JavaScript based on the more C#-like code typed in by the developer. The JavaScript generated is more complicated than what most JavaScript developers write and so may be a little harder for a thief to decipher after its been obfuscated and then de-obfuscated.

One last point to be made, and again, it’s not much, is that CSOM is going to be a smidge harder for a thief to reverse engineer than REST. This isn’t because of anything actually in the APIs themselves, but really just because of the nature of the traffic they put on the wire. REST is going to expose the explicit endpoints your code calls as well as the payload for the request and the response, typically in pretty easy to understand JavaScript Object Notation (JSON). CSOM on the other hand, is going to submit all calls to a ProcessQuery endpoint and the payload, while still either JSON or XML, is going to include additional information as well as internal object path references that can make things a little harder to decipher.

It’s not much, but really, it’s about the best you can do if you’re stuck on the client because of your SharePoint-hosted app. If this is the case, you pretty much need to face the fact that protecting your IP is not going to happen if someone is determined to get it. The best you can hope for is to make things a little more difficult and that maybe, just maybe, the thief will decide it’s not worth it and move on to someone else’s app.

Things are much better if you move to cloud-hosted apps because these give you the option of moving your important code to a server somewhere, which means that it is not accessible to a thief. In this case, either CSOM or REST is valid because the thief cannot access the code or the traffic over the wire so everything is pretty well protected. All they can get to is the end HTML/CSS/JavaScript which hopefully doesn’t have too much of your secret sauce baked into it.

If the application you’re building is not an app, but instead either a full trust SharePoint solution or else a rich client application, then things look a little different. In each of these cases, the decision between REST and CSOM is pretty minor because most of your IP protection is going to come by investing in a good DLL obfuscator and running your assemblies through it before distributing them. A good .NET obfuscator will make it impossible to open your assembly in something like Reflector and get at the raw source code. The only thing left now is the traffic over the wire. As with the SharePoint-hosted apps this is a concern because the thief can see the endpoints and the data for the request and the response. The situation here is the same – CSOM is going to provide a little (and I do mean a very little) extra protection. It may not be enough to make someone move on to another app, but it’s about all we’ve got.

Developer Concerns

As I said in the SPC session, I intentionally ranked this one last. It’s not because this isn’t important; it’s not because developers aren’t important (I’m a developer, so this impacts me too). I ranked this last because, as much as we may all hate to admit it, that’s where it really needs to be. The arguments we developers put forward for why we can or can’t do one API or the other don’t, in my opinion, hold much water. They tend to boil down into one of these:

  1. Lack of documentation (REST)
  2. Learn new things (REST & CSOM)
  3. JavaScript (in general)
  4. Lines of Code
  5. Professional Development Discipline

The first one is really the biggie. I hear this one pretty frequently – I don’t/can’t do REST because the documentation is bad. Unfortunately, it’s not true. As someone who has largely switched from CSOM to REST as my “default” API, I can tell you that between the official documentation and the hundreds of blog posts on the REST API in SharePoint or REST in general, there is plenty of information out there. Is 100% of the REST API documented? No, but neither is CSOM. There’s enough REST documented, however, for you to figure out how to do anything you need to do. The other thing about this one is that it is not a permanent problem. The documentation is getting better, between official sources on MSDN and unofficial blog posts where someone figured something out and wrote about it.

The second class of arguments is a little odd. If you don’t want to learn new things, you really ought to consider a new field. Technology changes too rapidly to be stuck in one place. I’m not saying that you need to jump at every new shiny object that comes along, but you ought not to shy away from something just because it is new. Interestingly, I hear this as an argument against both APIs.

People like to complain about JavaScript. While this isn’t strictly an argument for or against a particular API, it does get thrown out as a reason not to do client side development – JavaScript isn’t “real” code (whatever that means), JavaScript is too fragmented/loose/hard/insecure/whatever. Fine, I get it, JavaScript is new, JavaScript might be a totally different paradigm from what you’re used to. But that’s no reason not to learn it if that’s what you need to do. I railed against JavaScript for years a decade or more ago, but finally realized that this was a tide there was no way I was turning back and I had to either embrace it or pick a different career.

Lines of code is an interesting one because it is so easy to prove or disprove, depending on your point of view. I’ve found, though, that even when presented with facts, people still don’t change their argument. Quite often I hear people complain that REST requires too many lines of code to accomplish any given task, and that they won’t use it for this reason. I wonder about how they are determining lines of code when I hear this. I wonder why the number of lines of code required for something matters. Now, certainly, if we’re talking really significant differences here then certainly that matters. But if one API takes 5 lines and the other even 15, 20, or 30, who really cares?

I do generally hear this as an argument not to do REST as it requires more code to accomplish any given task. However, in my testing, and I showed this in the session, the exact opposite is true. For the six common tasks I used for my performance testing (listed above) REST took less than half the number of lines as doing the same thing in CSOM, provided that you are smart about writing reusable libraries (for each API, to be fair). A well written utility library can encapsulate the common, repetitive tasks from each API and not require you to retype them each time. Once this code is in a utility library, it should no longer count in your “lines of code” tally. My argument for excluding it from your count boils down to – do you count jQuery lines when calculating lines of Code? You probably don’t. And that’s correct. You shouldn’t. Once code is a part of a well-crafted, reusable library, it is no longer directly maintained for the project at hand and so no longer counts.

Last but not least in this area is “professional development discipline”. I’m a huge fan of architecting and coding in a good, professional manner. I’ve been preaching that message for years. But neither REST nor CSOM leads to bad code or bad coding habits by their very nature. Can you write bad code in either one? Sure. Is it easier or more prevalent in one API over the other? No. This is another one that I chalk up to just us developers being cranky.

So before you think I’m a masochistic developer who likes to make things hard for myself, let me assure you that I am not. I don’t like to work any harder than I have to. I do, however, like to use the right tool for the job, even if it means abandoning something that has worked for me in the past and taking on something new if it is the right thing for the task at hand.

[OK, I’ll jump down off my high horse now. Sorry about that…]

The Future

You’ll see a fair amount of stuff on the web saying essentially that “REST is the future.” While I won’t argue this fact, I think that it is, it is also true that for SharePoint, at least, Microsoft is committed to CSOM. I mentioned this before but it is worth repeating – CSOM isn’t going away any time soon. If you’re doing CSOM, or thinking about doing CSOM, don’t worry. It’s not a dead end.

Just realize that it’s a SharePoint-only technology. If you need to reach outside of the SharePoint world, you’re almost certainly doing REST for that. This means that there’s a pretty good chance that REST will be part of your repertoire.

The Final Verdict

In the end, I took a cop-out in my session and declared both APIs the winner – it’s like Kindergarten, everybody wins. Some people, I think, were a little disappointed that I didn’t crown one API the winner and declare that you should use it for all remote SharePoint development from now on. I don’t think it’s that cut-and-dry, though. They’re both perfectly valid in certain situations and you need to make up your own mind. I’ve given you the things to think about here when you need to pick an API, now you need to make up your own mind.

One final point here: there is no reason you actually have to pick one API over the other! They can coexist very nicely. I know people who use REST for reads and CSOM for Writes, Updates or Deletes. I know other people who use REST for interacting with SharePoint data, but CSOM for interacting with SharePoint constructs – Lists/Libraries, security, Sites/Webs, users, etc.

There is no reason you actually have to pick one API over the other! They can coexist very nicely

My Decision

Based on everything presented here, and my experience with each API, I generally prefer REST over CSOM. My default position is to use REST unless I can’t. That doesn’t mean I don’t use CSOM. I do, but it’s maybe 10% of the code I’ll be writing moving forward. As I said, I used to do a lot of CSOM, but I’m transitioning. This should have absolutely no bearing on your decision. I hesitated before putting it in here or including it in my session, but in the end decided it was useful to include.

Typically, I use CSOM in these situations:

  • Managed Metadata
  • Workflow
  • High volume/Low bandwidth situations (where every byte on the wire counts)
  • When a client is already invested in CSOM and doesn’t want to introduce REST

Closing

That’s it. That is pretty much everything I covered in my SPC session (minus the juggling). I hope you found this valuable. If you want to see the session and watch the demos (but you still can’t see the juggling), the recording is available here: http://channel9.msdn.com/Events/SharePoint-Conference/2014/SPC423.

If you want the source code from the demos I did, it’s available here: http://1drv.ms/1Smdumi 

What do you think? What factors do you consider when choosing an API? Sound off below, but keep it civil…

Speaking at SPC14

I’m happy to announce that I’ll be speaking at the SharePoint Conference in March. My session is Deep Dive: REST and CSOM Comparison, which fits in nicely with the Pluralsight course I made that just went live (and it’s successive parts coming in Q1).

Here’s the synopsis for my session: SharePoint 2013 is fully embracing the world of client-side technologies, both on-premises and in the cloud. This means JavaScript; but which API should you use – REST or CSOM? This session will briefly introduce each, and then quickly move into a demo-heavy comparison of both approaches, citing the pros and cons of each API. By the end of the session, you’ll have a solid understanding of when and why you would use one over the other (and some situations where you combine them into a hybrid approach) as well as plenty of examples of each in action.

 

Details available here: http://www.sharepointconference.com/content/sessions/SPC423

 

I’ll post more information here as the conference gets closer.  See you in Vegas!

My Pluralsight Course is Live!

It’s been a long time coming, but my first “real” course for Pluralsight is now live*. The course is “Developing SharePoint 2013 Solutions with JavaScript” and is available here: http://pluralsight.com/training/Courses/TableOfContents/developing-sharepoint-2013-javascript

This is actually the first of three parts to the course, with the next two parts coming in Q1 2014. Here’s the table of contents for part 1:

  1. JavaScript Fundamentals for SharePoint Developers
  2. SharePoint and JavaScript: Getting Started
  3. Core JavaScript Object Model Programming
  4. Reading/Writing SharePoint Data with the JavaScript Object Model
  5. Programming Security in SharePoint with the JavaScript Object Model
  6. Fiddler and an Introduction to REST
  7. Reading SharePoint Data with REST APIs
  8. Creating, Updating and Deleting SharePoint Data with REST APIs
  9. Deploying JavaScript Libraries
  10. Downloading JavaScript Libraries
  11. Common Third-Party JavaScript Libraries
  12. Final Thoughts for Part 1

Altogether, part 1 here is just over 6 hours long. Parts 2 and 3 add on an additional 9-10 hours for a grand total of over 15 hours of SharePoint JavaScript goodness by the time I’m done. I’ll post the outlines for the next 2 parts soon.

I’m obviously a bit biased, but I think there’s a LOT of good material in here – including some that I don’t think you can find anywhere else, or that other published content is just plain wrong or incomplete. It intentionally starts out very basic but then quickly moves into the deep end of the pool, covering both REST and JSOM. There are also modules on the best/proper ways to work with libraries (both deploying to SharePoint and downloading to the user’s browser) and finishes with a discussion on the REST vs. CSOM debate and which one you should use.

Stay tuned for more highlights on the content. If you don’t have a Pluralsight subscription, you can get one for $29/month, or wait for the companion e-Book (more on that later!)

 

*: I say “real” because there is a course I recorded for Critical Path Training originally that was converted to a PS course but I don’t really count that one

Security, Permissions, AllowEveryoneViewItems and Script Libraries

I came across something interesting today that I’m still mulling over; I haven’t gotten comfortable with this yet and so don’t really know how I feel about it and its implications. What I’m talking about is a property on SPList called AllowEveryoneViewItems. MSDN has the following to say about this property:

Gets or sets a Boolean value specifying whether everyone can view documents in the document library or attachments to items in the list. The AllowEveryoneViewItems property does not apply to all list items, but only to documents in document libraries or to attachments in list items. This property only works when users browse directly to a file through the browser, and it has no effect on the rest of the user interface or the object model.

The net effect of setting this property (either declaratively in your ListTemplate Elements file, via PowerShell or programmatically) is that you can make files in a document library or attachments on list items available (read-only) to all users, almost completely bypassing SharePoint security. I say almost because it does not, in my testing, enable anonymous access to the files. But if the users have any access to your site, they have access to all files in any list with this property set to true. If you turn on anonymous access for the site, then everyone has read access to these files

On the one hand, I see this as a huge boon for something like a script library – where we’re storing JavaScript files that should be available, read-only, to all users and we don’t want anyone to be able to turn that off or block access because then bad things will happen. On the other hand, I don’t like anything that bypasses SharePoint security. Security has always been a strong selling point for SharePoint in my mind so this concerns me a bit. At the very least, there should be some indication on the permissions page for the list or library (and the item if item-level permissions are in use) that this property is set to true and therefore everyone has read access.

What do you think? Is this good or bad?

Dave