This post was first published in April 2017 and has received several facelifts as our process has evolved.
Stages of Question Development
Before we get to a real example, it will help for you to understand the high-level process. Every question in our system goes through five stages:
- Research. Every question starts with a regulation and sometimes there are multiple regulations that drive a question. The reverse is also true – sometimes there are regulations that drive multiple questions. In either case, it’s our job to read and interpret those regulations to determine where there is a compliance need. For local regulations, the additional step of reviewing the state level regulations to determine areas of overlap becomes necessary.
- Draft. Once we identify a compliance need, we write a question to address that need. Question-writing is a difficult combination of art and science. On the one hand, the question must be written in “layman’s terms” and in a way that it can be answered in a yes/no format. On the other hand, the question must also be written in a way that it remains true to the regulation while not putting Simplifya into a position where it could be accused of a UPL violation (Unauthorized Practice of Law). Note that the question and its associated sub-questions and action items are all drafted at the same time.
- Quality Assurance. The QA stage is the most time-intensive. We of course do the expected proofreading to catch spelling and grammar errors, make sure the correct citations are listed, make sure we’re consistent in terminology, and other details. But the QA stage is also where we identify opportunities to make the audit process easier for our clients. We do this by eliminating certain questions that are made redundant by the selection of certain licenses, grouping questions into logical categories, considering the use of “applicabilities” to reduce the number of questions that need to be asked, and so on. It is common for questions to go through several rounds of QA.
- Release. At some point, the question is deemed suitable for client use and released into production. This is a very subjective decision, but we’ve compromised on the idea that when we think it’s as good as or better than the competition, we release to production. We can’t know this of course, but we’ve been getting anecdotal feedback that gives us a general idea. This is how we honor one of our core value principles to have the best regulatory content. Today, the Release process is more cumbersome than we’d like. The legal team generally does all of the drafting and QA in Word to take advantage of its tracking feature. As the release date approaches, the content is moved into the system via a clunky combination of copy-paste and uploading via CSV.
- Refinement. Internally, we have a higher bar than “as good as or better than the competition.” The Refinement stage is that higher bar. This is where we feel we have an opportunity to refine our content in ways that add value to our clients. In some cases, we’ve implemented software updates that allow us to modify question flows or even the number of questions that are necessary. In other cases, the regulations are changing in ways that force us to change the questions and associated Action Items. And because the questions are already in production, we may also be incorporating feedback from clients and partners. This stage continues until the question is made obsolete, either because the regulation changed or we found a better way to handle the compliance need.
Conclusions
This Content Development Life Cycle (CDLC) is similar in concept to what many know as a Software Development Life Cycle (SDLC). The SDLC has been around for several decades now so its value as a planning tool is undisputed. We can apply many of its principles to our CDLC to make some observations and draw the following conclusions:
- When creating a checklist, the bulk of our pre-production time – about 44% – is spent in QA. If you hear Marion talk about how we underestimated the time required to create a checklist, he’s talking about the QA stage. It’s relatively easy to write a list of questions. It’s exceedingly difficult to make sure those questions are displayed and hidden based on their application to the business (i.e., it’s licenses and types, jurisdiction, number of locations, etc.), presented in a logical order, in a manner that all employees can understand, and remain true to the spirit of the regulation.
- We can accelerate our expansion to other states by partnering with companies that have regulatory expertise in those other states, but their contribution to our CDLC is limited to the Research and Draft stages – about 50% of the pre-production cycle. We still need to handle the QA and Release stages. Once the question is in the Refinement stage, the partner is expected to stay on top of updates and we will still handle the QA and Release for these updates. Currently, we have one Certified Partner in New Mexico.
- For perspective, the cycle for the Colorado checklist was close to 1,000 hours and we are still making significant updates through the Refinement stage! We didn’t realize it at the time, but Colorado’s regulations are the most comprehensive and complex in the country. Part of this is because we are one of the few states that legalized both medical and recreational marijuana use, but Washington is also med/rec and their regulations are just one-third the length of Colorado’s. So while the Colorado checklist has been a bitch to create, we’re proud to have conquered it first and encouraged that the road going forward will be smoother.
Super interesting, right?! We could ramble on about the CDLC for another 1,000 words, but let’s stop here. Now you have the context that will make part 2 a much easier read.