Loading JavaScript Libraries – It Shouldn’t Be This Hard

I’ve been doing a lot of JavaScript programming for the past few years. SharePoint 2013 has only accelerated the pace. Currently, I split my time roughly evenly between SharePoint and non-SharePoint work at my primary client, but both projects make heavy use of JavaScript, between them making use of many of the popular JavaScript libraries – Angular, JQuery, JQueryUI, Moment, Require, Knockout, etc., etc., etc.

The disparity between the two worlds is still quite wide. Even something as simple as loading up required libraries is more difficult in the SharePoint world than in the rest of the world. Part of this, I realize, is due to the nature of SharePoint. It is a much larger application than many other non-SharePoint applications and lends itself to extensibility much more than those non-SharePoint applications. There are often far more people involved as potential customizers of SharePoint than with non-SharePoint applications

And that’s where the trouble begins.

Extensibility means a loss of control. Even in controlled environments that actually effectively do SharePoint governance, extensibility invites problems. JavaScript is still a little bit of the Wild West; every project is going mandate the use of different libraries, different versions and different configurations. If all I’m concerned about is my project, things are easy: I load the libraries I need using one of any number of script loaders or script-loading approaches. To hell with anyone else.

That, obviously, is not a very good approach, but it is one taken all too often. I was recently asked to look into a problem at a client that boiled down to them having a ridiculous number of JavaScript libraries loaded on too many pages:

  • 4 different versions of JQuery
    • One page actually loaded JQuery 6 times, as poorly trained site admins, or users with permissions, were adding Script Editor Web parts on pages and each loading JQuery, from different document libraries and CDNs
  • 2 different versions of JQueryUI
  • 2 versions of Knockout

This is in an environment that at least makes an effort at governance. Unfortunately, too much of it is left in the hands of developers, but this isn’t a post about governance directly. Instead, it is an attempt to try and figure out what we as developers and architects should do.

As a consultant and ISV, I cannot mandate too strictly how my client or user’s environment be configured. I can say something like – if you don’t have JQuery installed and loaded, we will install and load it. But that’s about it. That may be fine for the moment my customization (either custom built as a consultant or a product I sell as an ISV) is added to the environment, but I have no control over what happens later. If, three days after my customization is added, another product or customization is added which loads its own version of JQuery we may have a problem – especially with the somewhat recent split of compatibility between JQuery 1.x and 2.x. The one that prompted me being called in to that client was caused by a customization loading a new version of JQueryUI, one which contained a different custom collection of widgets. It broke a different customization.

What’s the solution? I don’t have a good answer, but I’m working on it and invite collaboration. If we can come up with something good, this would be a killer community-driven project. Here are the problems, as I see them:

  1. We need some way to track which libraries and versions are available in an environment. This can’t be “just” documentation or offline means – code should be able to check and respond appropriately. Somehow, this needs to be global across the environment and future-proof.
  2. MDS must be supported. As a consultant or ISV, I cannot dictate that MDS be turned off and have that remain inviolate for all time. I know many developers prefer to just turn it off, but it is a feature of SharePoint and therefore must be supported.
  3. There must be a standard for how libraries are loaded so you can anticipate when they will be available. One of the problems faced by the client I mentioned was that via the convoluted mess they had gotten themselves into, JQueryUI was loading before JQuery, which doesn’t work.
  4. There must be a simple way for non-developers to participate in maintaining things. Otherwise we run into the problem above where users had loaded many different versions of JQuery via the Script Editor Web Part
  5. It would be nice if this made things easier for developers – perhaps by supporting AMD modules and requirements management, either via Require.JS directly or a fork of it to make it SharePoint and MDS compatible.
  6. It should support both on-premises and SharePoint Online/Office 365, including Apps.

There are a few other things that should probably be included in there, but these, to me, are the big targets. I realize that there are answers to each of these individually, but what is missing is a comprehensive approach and a commitment from the community to drive a consistent approach. Honestly, there may not be one ring to rule them all. The best thing may be two or three approaches, but that would still be better, especially if there were some consistency and commonality amongst them.

If anything is to come of this, education and community support are critical to making it work, so what do you think? Leave a reply and we’ll see where this goes.

Dave

14 thoughts on “Loading JavaScript Libraries – It Shouldn’t Be This Hard

  1. Marc Anderson

    David:

    Great topic! As you might expect, I’ve been wrestling with this for years now, given the way I develop for SharePoint.

    Unfortunately, I don’t think that there can be a fully technological solution to all of this. I don’t think that’s just because it’s JavaScript either, but the fact that many people can drop JavaScript into a page does make it more complex. So I think it’s a combination of technology and governance that makes it possible to succeed.

    First off, as developers, I think we need to be offering JavaScript libraries as a service. I wrote about this in my chapter in “Black Magic Solutions for White Hat SharePoint” a few years ago. We should be offering things like jQuery, jQueryUI, etc. to our organization as a service, informing them clearly what is available and what the rules are around using them. This ought to include functions that are useful inside our organization, too. For instance, we might have a JavaScript function to look something up on a server on the other side of the LAN. We should make those interfaces public, just like we should with things like jQuery. I call this “FaaS” – Functions as a Service – in my chapter. We’re all going to be far more successful working *with* citizen developers than against them.

    Next, we need good tools to manage what gets loaded in any given page. I find SharePoint’s SOD and ScriptLink stuff overly complex and confusing and instead tend to use LABjs whenever I need to manage loading multiple JavaScript files where either the sequencing matters or the libraries might already be loaded. This doesn’t help much with MDS, but I’ve been successful building around that (not disabling it) anyway.

    Now, all that said, I prefer not to build Apps. I find that for the most part the things I build can be managed pretty darn well without the complexity the whole App model and its auth convolutions introduce. We do need to consider, though, that everyone who builds Apps decides how to load *their* JavaScript dependencies, and it’s all over the map. We’re as likely as not to see multiple loads of the same library coming from using more than one vendor-supplied App as anything else.

    None of this is simple when you look at it holistically, but in each case I do think we need to decide what levers make the most sense. There is not, nor there should be, a one-size-fits-all solution.

    M.

  2. David Mann Post author

    Marc:

    To respond to a few of your points:
    I don’t think that there can be a fully technological solution to all of this: I agree totally. Education and governance have to be a part of this. On the flip side, though, it isn’t all non-technology, either.
    Functions as a Service: Nice term. I’ve been advocating that for a long time, and have some rudimentary code/prototypes around somewhere to support this from a technology point of view, so I have some ideas on how to make this work.
    Tools: Also agree. SOD and ScriptLink are too difficult for non-developers to grok completely. Unfortunately, I think LAB probably is, too. If we’re supporting business people writing code, it needs to be simpler, not because they’re not smart enough, but simply because they’re not trained and will tend to take the easiest path (gross stereotyping there, yes). If we’re going to forge a successful approach, it needs to be that easiest path
    Apps: I feel the same way about Apps as I do MDS – it’s a part of the product so can’t be ignored. While there may not be a lot of non-devs building apps, the conflict comes in when you have devs doing one thing and non-devs doing something different. They are going to step on each other’s toes.
    One-size-fits-all approach: I think we’re in agreement here, but just to clarify – I think there is one solution, it may just have a number of parts or approaches under it’s umbrella, but it is all a part of a holistic whole. It has to be to be successful.

    I have ideas on how most of this can be made to work, but I don’t want to share them until other folks chime in a little so as not to poison the well. I don’t have all of the answers and don’t want this to be a one-person effort. To be successful with what I would like to see, it has to driven by a broad base. Perhaps I’ll work on a follow-up post that lays out the problems in a little more detail so folks can see more about where things are out of whack.

    Dave

  3. Hilton Giesenow

    What an awesome discussion! David, I’d suggest that SPO be included as part of the “part of the product” discussion – it’s easier as part of the “FAAS” solution if you own the environment and at least have the option to deploy to Layouts, perhaps, but it’s not required of course and maybe there’s a way to suit both solutions well – something like a content type hub, but for JS? It would work fine in both places, and it’s also something that’s migrate-able.

  4. Simon

    Been wrestling with this for ever. In our 2007 installation I created a Central Admin thing that let you add script links into a Delegate on the masterpage. So we had control over what JS libs we included by default. Of course we told people but we still have the problem of someone copying Marcs code 😉 and just pasting it in and then it starts. Even our most experienced Content Manager forgets and all of a sudden pages have duplicates and inconsistencies.

    Just wished Microsoft had tried to tackle it somehow, at least from them its in all the documentation and on everyone’s blog afterwards, if they had done something then every script example could point at the how to include script articles instead of everyone just including them in the example code. Until MS grab this one by the horns we’re always going to be battling with it.

  5. Marc Anderson

    Wouldn’t it be great if there were a setting per Site Collection like the alternate CSS one, but for JavaScript? The more they want SharePoint to be a service, the more they need easy to use configuration pages to set stuff like this. We shouldn’t have to write so much code to do something as simple as saying “include jQuery in the master page”.

    And sorry about all that code of mine that people are pasting, Simon.

    M.

  6. David Mann Post author

    Marc: There *could* be a per-Site Collection configuration setting. That’s one of the things I’d like to see happen if this discussion turns into a community project.

    Hilton: SPO is definitely “part of the product” and anything that happens will need to support SPO/O365

  7. Hugh

    I thought I would chirp in and give my insights, but firstly I want to reaffirm there is a problem, but there are solutions.

    Unfortunately all great solutions require some code to start, but I can see ways of simplifying it if we need users to be able to use external libraries.

    Case study
    Let me take jsfiddle as my case study. Asking for a library is as simple as selecting it from a list. Then you can add your code snippet. This is an incredibly useful pattern and we can learn from it when we have to architect this scenario.

    Example implementation
    First any way of adding scripts to a page should be blocked from SharePoint. No Script tags allowed, no wsps with custom actions etc. Then we could introduce a new simple webpart in it’s place. Take the Scripts On Demand code, and wrap it, and have a list that goes with our webparts for usable JS files. The end users can then simple add the webpart, and use it easily configurable. Our governance rules can check the code, and accept/reject it for running, or even submit it for approval.

    Closing
    Scripts On Demand and MDS aren’t easy to work with, they have their own strategies and complexities that are EASILY simplified (I recently wrote a blog post demonstrating this).

    Now while we are ‘doomed’ into writing such implementations and rolling our own governance code for now, we have to also remember different companies have different governance rules.

    Including JavaScript willy-nilly however is bad governance, so the practice needs tying up badly.

  8. Pingback: Loading JavaScript Libraries–It Shouldn’t Be This Hard « Aptillon Blog

  9. Paul Tavares

    At the root of this problem, in my opinion, are two things:

    – Following good development practices and guidelines
    – Adapting the use of modules

    The problem highlighted here is not unique to SharePoint. I recently was asked to make some improvements on a ThoughtWorks Mingle system which could suffer from the same issue detailed above if too many other “customizers” are adding their plugins/apps/code/web parts, etc… I suspect that WordPress and Drupal also suffer from such issues. Too often I watch users of SPServices post issues that say “SPServices” is undefined… normally because they have multiple versions of jQuery loaded.

    Anyone adding customizations to SharePoint (even if a copy-and-paste) should really be treated as a developer… I don’t care that you are a project manager that saw something on a blog post and thought it was a good idea to copy the code to a SharePoint document library… there are implications that you need to be aware of and understand before doing that… (Easier said than done – I know).

    Good Practices:
    One of the number one rules you see just about in any good tutorial is: DON’T POLLUTE THE GLOBAL SPACE! if this was followed, then the problem around supporting MDS would not be an issue… As it understand it, MDS simply clear anything attached to the global space when a new page is “loaded”. This very simple rule would solve allot of problems…
    Once a person understands how to manage context, they quickly embrace more sophisticated tooling that enables them to create self-container/non-conflicting apps/customizations… Mark is dead on in that FAAS is the approach – the web standard (ES6 or ES7 – I forget) calls them modules… This concept can be used today by defining functionality in AMD modules that are then loaded as dependencies in the browser. I recently converted one of my SharePoint open source project to AMD, because I wanted the flexibility of using only what I needed… I now use AMD on all my projects and will eventually switch over to the standard when that is widely supported by browsers.

    Multiple Versions of a Library or same version loaded multiple times:
    RequireJS is the commonly used library to provide AMD loading and with the use of namespaces, the problem of loading multiple versions of the same library or the loading the same version multiple times would be addressed. That being said: This will not always address the need to have a truly private version of a library… There are still cases where having a private version is legitimate and one I use very often (ex. attaching private plugins; using more recent versions of other plugins, etc).
    The concept of avoiding the loading of the same library multiple times can still be addressed even without RequireJS, but it puts the emphasis on the user. In cases where I want to re-use certain libraries, I not only check that they are already loaded, but also check what version they are… I only load my own version if one of those does not meet my needs – but I always ensure that the version I load stays private and does not impact any version already loaded. Again: this is on the user… and we all know that means (its not going to be followed).

    Tooling:
    I suspect that most companies would not want to advertise the existence of a “standard library loader” to general users… since that would implicitly suggest that ad-hoc customization is ok…maybe even supported… But if one is needed, then my suggestion is to NOT re-invent the wheel and rather use requireJS for this purpose… Multiple versions of libraries can be defined in its configuration allowing those providing customizations the ability to “require” the version they need… RequireJS ensures that a module is only loaded once and, through the use of namespaces, ensure that each “app” receives a unique instance of that module. Example:

    define(['jquery-1.11.1'], function($){
    // $ = jquery 1.11.1
    alert('jQuery Version: ' + $.fn.jquery);
    });
    define(['jquery-2.0.0'], funtion($){
    // $ = jquery 2.0.0
    alert('jQuery Version: ' + $.fn.jquery);
    });

    There would need to be some tweaking in how RequireJS remains loaded in a MDS environment, since it registers itself globally at first.. This is maybe the area where a good wrapper could be created. One that would be used by customizers to get a reference to requireJS… There are ways to “hide” a global reference to a library that would survive the MDS global space cleanup.

    Tooling is great, but only if there is adoption… That’s where the community and specially the influencers within the community come into play (I’m looking at you MVPs :) )… If this was to talk off, then any code sample we post on our blogs/forums/gists/etc… could incorporate the usage of such tool… eventually, this will triggered folks to start using it, but better yet: it takes care of you “copy and pasters”…
    One last very important note about this “tooling”: it should NOT require any type of Server side solution or for any sort of Site Collection configuration… All of this can and should be done client side. Remember: users are going to be users… if its not easy (ex. just include an additional library with my code) then they are not going to follow it.
    And BTW: this type of tooling does not even have to be hosted in a SharePoint site… Should it become popular enough, it could be placed in a CDN and included in a page…. A few lines of code in a user’s customization would invoke it.

    Re: Apps
    I know everyone associates that word with SharePoint’s v2013 introduction of the term and blotted process that comes with it… the truth is that Apps have been created all these years, using web based technologies… these “apps” live in the browser and are installed by copying the needed files to a document library…

    Wow… that’s allot of writing… sorry… :(

    /Paul Tavares

  10. Matthew Carriere

    So my biggest takeaway reading this is that the JavaScript development isn’t being treated like a first class development language.

    Here’s what I mean.

    Tracking different libraries

    Having multiple libraries is only a problem from a performance standpoint. You should try not to be loading 4 versions of jQuery, but if you are namespacing your jQuery files then it shouldn’t be an issue. I would agree the namespacing presented in the comments is complex, it also requires oversight – which you might not have across a large SharePoint environment. As a developer you should be working to make YOUR js libraries isolated from anyone else. A simple way is to redefine your jQuery variable right after its declared from say $ to owl$ (what we do) or NWF$ (what nintex does).

    MDS must be supported

    I would agree. If you’re turning off sharepoint functionality to support some customizations you’ve made – your doing it wrong.

    Load Order

    There are several solutions here. Users don’t need to understand my code or its architecture. Development is not simple, front-end development is no different. A developer should have no problem learning how LAB works and using it. However there IS a simpler approach. Generally for performance reasons you would bundle all you js into a single file anyway (let’s call it applicaton.js) basically, if you concatenate your files into that one file, in the order you want. Problem solved.

    There must be a simple way for non developers to maintain things

    Let me be clear. NO. This is development! No a non-developer should not be able to understand my code and maintain it.

    If this made things easier for developers

    We are doing front-end development that has to port back to IE8 on top of SharePoint with all of its existing front-end functionality we can’t break and – other vendors will be developing solutions as well – no part of this is going to be easy. Building front-end solutions on SharePoint is about as hard as it gets.

    It should support SharePoint Online/Office 365/on-premises and apps

    This is basically why you would want to develop a front-end only solution, however you are still going to run into differences. Those differences aren’t bad. They are fundamentally different and will require maintain 2 code bases and properly pushing changes into each as necessary.

    So to reiterate. This is front-end development, this isn’t end-user customization using JavaScript. What’s required is to think hard about which framework you are going to develop you solution in, examples, angular, ember, backbone (because YES you need to pick one and define how your architecture will look). Then you need to have a strong developer toolchain, this is source control, issue tracking, builds and testing (which by the way is made MUCH EASIER with front-end development in SharePoint 2013).

    Agree with Marc Anderson completely. There is not going to be a one size fits all solution. There are well crafted solutions that take careful thought through the design and development stages of its creation. As a community we need to know what parts of the toolchain work best for what we are building and how to not conflict with existing solutions and functionality.

    It not easy.

  11. Scot Hillier

    If you control the environment, then a simple answer would be to create an app that manages references to common libraries and injects them using the PnP script injection pattern.

    http://channel9.msdn.com/Blogs/Office-365-Dev/JavaScript-injection-in-SharePoint-Online-Office-365-Developer-Patterns-and-Practices

    You could easily include references to common CDN libraries. Then deploy your own stuff as a CDN and add it to a list in the app. This would inject the library references on a per web site basis, which is pretty good governance. Although obviously not perfect.

    1. Matthew Carriere

      The PnP script injection pattern is good for a trivial use case, maybe throw jQuery in there and run something afterwards, but its way too slow and too late for anything substantial. If you use it to load angular and pull a bunch of data, show that data etc. Your net effect will be – open the page, watch sharepoint load, then you’re pnp happens, then your stuff loads. Its a jarring effect that end users will hate.

      The next thing you run into – as mentioned in this post – is load order. You have to run something like LAB to manage what runs when, all while doing it after SharePoint is finished.

      1. Scot Hillier

        I’ve definitely witnessed the “jarring effect”. But there are very few options. We can’t (well we’re not supposed to) change the master page. So, everything happens “after SharePoint is finished”. I looked through the threads again – maybe I missed it – but I don’t see another answer presented that deals with the fact that Microsoft says not to edit the master page.

Comments are closed.