(originally published in Best Practices, October 2015, Volume 17, Issue 5)

• Why test a CCMS?
• When is a good time to choose a CMS?
• How to test a CCMS?
• What are the requirements and criteria for CCMS?
• Choose the best-fit CCMS

How do you decide which CCMS (component content management system) to buy? Or have you already invested in one? Hopefully it was because your content team and collaborators have tested a few systems and committed to achieving their tasks and objectives with the system they chose. If that’s the case, let the techcom community know how you proceeded, because you are a role model.

Although DITA has been implemented around the world for more than ten years now and content management systems (CMS) and structured writing have been around for even longer, managing CMS projects still seems to fail at a considerable rate. Moreover, as the most recent CIDM survey showed, out of 91 respondents working with DITA for longer than 3 years, “despite the success, 62 percent report working to improve tool performance, and 37 percent report needing to acquire better tools. Providing additional training for authors is still a requirement for 38 percent.” [1]

As in any project, you have to take into account the risks at each phase, so it’s understandable that the implementation might take a wrong turn at certain points. Yet there are frequent articles and reports mentioning a rise in failed, faulty, or incomplete implementation projects at alarming rates, especially when they involve “changing mindsets and culture”.[2][3]

The risks are often due to human factors, politics, skills, budgets, and plans, a great many having to do with involving the real users of the future tools, while collecting the requirements and testing to select the proper system.

Why test a CCMS?

Let’s say you’re considering buying a family car. Would you let someone else, say your bank account manager, select a car and do the test drive for you? Would a test drive on a straight highway on a sunny day be enough for your decision, when you usually drive on hilly, foggy country roads?

Acquiring a CCMS is a long-term investment in a strategic asset–your user assistance content. Its implementation is a complex project, which requires major changes in processes, workflows, and most often in mindsets. Testing is not optional.

There are two important issues when it comes to testing that are usually overlooked:

• You have to test on real, relevant content. Tools often come with sample content for a quick show case, like the “Washing your car” or “Changing the oil” DITA topics we’re all familiar with, but these would never cover your own types of content. You need to collect representative samples from your actual documentation inventory to test the selected CCMS.
• You have to test with real players. Having one person clicking around the CMS interface and quickly checking the usual steps, like writing, editing, and publishing, does not mean the entire team would have the same experience and would complete the workflow in the same manner. Your TechPubs team may have members with different levels of expertise, and, most importantly, each person has a different way of learning and working with tools.

When is a good time to choose a CMS?

Timing is always of the essence. If you are implementing DITA, should you do the CMS implementation at the same time? Although a CMS is in most cases necessary and it would influence the architecture and workflows, the implementation of DITA and of a CMS are two separate types of projects with specific phases shown in Figure 1 that add to the workload of the same team. And by the way, they’re different than the Research > Develop > Test > Deploy phases IT Managers typically work with.

Often implementations are difficult because the stakeholders are facing three new concepts at once: structured writing, the DITA standard, and working with the new tools. Tool providers have seen the need for embedded support while learning and authoring DITA content. New and better author assistance features are being implemented with each version, but too often the content teams are given only tool training, without training in structured, topic-oriented writing (STOW) or DITA training. Such a “shortcut” approach is unfair to all parties, since there are many flexible ways to manage the implementation of standards and tools. Certainly, the standards and tools must be correctly understood, if writers are to master the new information-development environment.

Especially when your team is migrating from unstructured to structured, topic-oriented authoring, there are some challenges you have to address before selecting and implementing a CCMS:

• auditing and modeling your existing content, without specific content tools in mind
• analyzing the TechPubs team skills map
• training at least a “pilot team” to work with the new methods

If they are new to structured models and DITA XML, authors need time to learn, to rethink their processes, and to practice with DITA authoring for a while so that they can consciously collect their requirements for the “next generation” toolset and feel ready to assume ownership of the new system.

Why address all these issues before buying a CCMS? Because the implementation itself is not your goal. You are aiming to empower your team to deliver the best user assistance in an efficient, professional manner. You cannot declare the implementation a success as long as your content team remains frustrated, struggling with a poor author experience in yet another tool added to the chain.

How to test a CCMS?

Assuming we define a separate project plan for the implementation of a content management system, CCMS Testing could be considered a sub-project (marked with the little red star in Figure 1), containing three phases (as shown in Figure 2):

1. Selecting CCMS candidates
   • collect requirements
   • prioritize
   • research the market
   • analyze and select the top candidates
2. Testing
   • configure sandboxes
   • select content samples
   • build pilot projects
   • define workflows
   • define test scenarios
   • perform tests
3. Evaluating and deciding on the winner
   • centralize results
   • evaluate
   • choose the best-fit CCMS

When collecting the requirements, do consider all the stakeholders. Don’t let it come as a surprise that IT and Legal ask for a higher security level for the information or that Engineering would want to do the reviews in their own repository.

To build a proper testbed (pilot project), collect representative information objects from your documentation inventory. Make sure you create all the types of objects and elements your team would use. For example, in DITA projects, you might need maps, submaps, keymaps, all topic types, topic collections, relationship tables, images and tables of different flavors, nested lists, equations, code snippets, notes and footnotes (hopefully you managed to get rid of those while auditing and modeling legacy documents), prerequirements, safety advice, choices, and so on.

Think of the entire lifecycle of the information and assign pilot testers to each role: authors, information architects, reviewers, editors, translators, and so on.

Remember: Test with real players, on real content!

As guidance for defining the content workflows, you can refer to the ISO/IEC/IEEE 26531:2015 standard (Systems and software engineering — Content management for product life-cycle, user and service management documentation), which defines the specific workflow stages (chapter 8), as well as quality management steps for each of the stages (chapter 10.1). In short, the typical content workflow stages are

1. Project initiation
2. Content editing and proofreading
3. Content technical review
4. Content testing
5. Content approval
6. Content translation
7. Content publication
8. Content sustainment and reuse
9. Content archiving

If you want to start with a simpler workflow, you could consider the stages as outlined in Figure 3.

The Research and Modeling stages are usually done outside the CCMS, but the content writers and collaborators have to understand that these stages are not optional and they may even take the longest time to complete.

Normally, the “state of content” would be a multitude of such workflows in different stages at a certain point in time, assigned in parallel to multiple roles and actors. When you start planning the test cases, you should define one basic workflow and build on that.

The structure of a test case doesn’t necessarily have to be complex, nor does it need special tools. You can use a spreadsheet specifying the actors, the content source, the actions, and the check points with expected results. Figure 4 is an example of a simple test case. As you go through the content workflow, the test cases may be longer, with variable numbers of actors and actions.

But what should your test cases cover? You’ve probably heard the joke about the tester walking into a bar… he orders 1 beer, 99 beers, -1 beer, a crocodile, a qwe123… .His is the testing attitude: draw your test cases to make and break the typical and the not-so-typical workflows. Test the regular tasks, like authoring, reviewing, and editing, but also plan to test the less frequent tasks that occur in a release cycle: rename and delete folders and files, schedule publishing jobs, edit stylesheets, try versioning and branching. Test the administrative tasks as well: user management, reports, filtering the dashboard, import/ export sources, and output.

What are the requirements and criteria for CCMS?

To centralize the outcomes of the testing, you can use a series of spreadsheets similar to the one suggested by the VDMA (the German Engineering Association) [5] in Figure 5.

The evaluation matrix contains feature criteria such as frontend usability, topics management, versioning, administration, and so on, as well as cost criteria, like licensing, maintenance, and so on. There are detailed charts for each feature. Adjust the list according to the requirements you have collected and prioritized in the first phase.

Requirements and recommendations for CCMS are also described in the ISO/IEC/IEEE 26531:2015 standard (chapter 12) [4]:

12.1 CCMS framework

12.2 CCMS management

12.3 Content object and asset management

12.4 Graphics and multimedia management

12.5 CCMS administration

12.6 Content authoring

12.7 Workflow

12.8 Content publication

12.9 Localization and translation management

12.10 CCMS interoperability

Choose the best-fit CCMS

So you have prioritized requirements, run tests, and evaluated the CCMS candidates against the criteria, but to identify the best-fit system, you have to make sure that it corresponds to the team’s skills and that the implementation and use are realistic and efficient. In the end, it is not the most complex nor the least expensive CMS that wins. You need to find the balance and make sure the system is going to be used as intended (see Figure 6).

It may take a couple of years to implement DITA and a new CCMS, yet this is only the beginning. The real work is just starting after the implementation, when the entire team is on board, using the system as intended, creating new projects and migrating legacy content.

As the CIDM survey mentions [1], if some DITA adopters are tempted to blame the tools for the “productivity paradox,” they should look deeper for the real causes of failure, like “shortcomings in change management” and insufficient investment in training, where the authors do not feel confident they can “design what their customers need”.

If TechPubs teams are committed to learn and if they enjoy a better author experience in the new environment, they will provide the best user assistance. Only then can you look forward to the return on investment you have planned for the implementation of the new content management system.

Reference:

[1] JoAnn Hackos, The Productivity Paradox, Best Practices, December, 2014, Volume 16, Issue 6

[2] IBM Global Study: Majority of Organizational Change Projects Fail–Changing Mindsets and Culture Continue to Be Major Obstacles, Retrieved 14 October 2008, from https://www-03.ibm.com/press/us/en/pressrelease/25492.wss

[3] “Why Projects Fail” blog, Facts and Figures, Retrieved September 7^th, 2015, from http://calleam.com/WTPF/?page_id=1445

[4] ISO/IEC/IEEE 26531:2015 Systems and software engineering — Content management for product life-cycle, user and service management documentation, Published on 2015-05-15

[5] Einführung eines Redaktionssystems für die Technische Dokumentation, 2010, VDMA Verlag, ISBN 9783816305934