Throughout the Application Security space, we’ve noticed many training "solutions” are not intuitive enough and don’t provide the engagement needed to stimulate and upskill developers.
Like tournaments, assessments are part of the puzzle that forms the Secure Code Warrior® training program. They’re an excellent way for developers to check-in on their skills and receive valuable feedback on how they can take themselves further.
We’ve put together some tips on how to maximize engagement levels with assessments.
Things to Consider
Assessments are a stepping stone towards knowledge retention and engaging training. They're not intended to be an exam or cause stress. It’s a chance for developers to really analyze their skills and find out what their strengths and areas of improvement are.
These kinds of opportunities don’t come along often once a developer is in full-time work, so it’s important to focus on the upskilling benefit.
Let your teams know this is an inclusive culture built to communicate goals and measure progress.
When thinking about how to communicate and encourage your teams, consider:
- Reminding everyone that assessments are not a test or part of performance reviews
- Fun, engaging messaging around assessments - what do your devs get out of this?
- Competitive rewards for assessment completion
- Peer to peer activities to do with the importance of secure coding best practices
- Having Security champions - passionate about security, training, and supporting others
- Hosting a Q&A session - What challenges are your devs facing today? What do they know already?
These are just a few general tips to keep in mind when you’re building out your assessment processes. We focus a lot on branding and theming your security training program, but that's because it's helpful. It keeps program visibility high which leads to better engagement and more general curiosity around what cool things your development teams are a part of.
It’s good to create a bit of a process around assessments so you can share the right message and ensure your teams are feeling the beneficial aspects.
- Reward developers for seeking out and completing assessments
- Create achievement certificates like a physical (or digital) acknowledgment of assessment completion and your team’s willingness to learn
- Send regular email reminders about the importance of training
- Remind teams that support is available via Team Managers or Security Champions
Depending on the type of assessment, you can structure things a little differently.
1. Keep the assessment manageable for the information being asked of developers.
Below is a high-level example of a schedule:
- First assessment - Top 3 vulnerabilities affecting your organization
(Approximately 1 hour. Include 6-8 easy and/or medium difficulty level questions)
- Second assessment - Remaining top vulnerabilities affecting your organization
(Approximately 1.5 hours. Include 8-10 easy and/or medium difficulty level questions)
- Following assessments - Build up to assessments featuring more challenging vulnerabilities
Two Important Notes:
- For baseline assessments with no retry expectation and no minimum pass percentage, it's okay to include more than 15 questions.
- If the assessment has a minimum pass percentage, it's always better to keep it smaller and more manageable so developers feel encouraged to try again if they don't succeed the first time.
2. Aside from baseline assessments, it's recommended to allow retries to encourage more attempts.
Tip: You can set the retry option to invite-only if you want to know who is interested in assessments
3. The minimum pass percentage should generally be between 70%-85%. This will depend on how challenging it is, what level of developers are being assessed, how other assessments within the organization are being measured, etc.
Example: For the same assessment, you may set it so that junior developers need 70% to pass while senior developers need 80%. Some customers have set the pass percentage to 60% for harder assessments, newer developer groups, or very junior developers.
4. The granularity with using subcategories depends on the goals of the program. If a subcategory truly plagues your organization, it's good to focus on it. Categories tend to cover the same high-level concepts, even if the subcategories are different, so it can be beneficial to ensure developers are upskilled across the category rather than focus on subcategories.
5. Time limits tend to vary on a customer by customer basis. You can leave assessments open for a few months, or decide to have everyone complete it on the same day. It depends on the expectations you have for your security training program.
6. Whether an assessment is open to self-assess or invite-only depends on where it sits in your security training program and how you'd like to keep track of when assessments are being taken. Having a few open assessments is a great way to let your teams measure their progress on their own initiative.
Start off by properly introducing your developers to Secure Code Warrior assessments and what they’re for. If possible, we’d suggest doing this by gathering all participants in one room. This will make it easier to address any questions or concerns that come up and keep all teams across the answers.
For companies with locations across the globe, it’s worth trying to organize something similar over a video call. Consider having your security champion, or someone that’s often sought after in your development community, help explain the importance of assessments.
Here are a few things to remember:
- Create assessments that are tailored to the development group taking them whether that’s a department, language/framework, etc.
- If you're having an assessment on a specific day, give developers a reasonable amount of time to complete it. 10 challenges can take over an hour, depending on the difficulty level
Basically, you want to build assessments that serve as a baseline for all your training moving forward. There isn’t just one assessment to cover all development teams, so you should expect to be creating a few of them at a time, depending on the languages your teams support.
As an Admin, look at what’s most important for your teams to achieve. How do you envision your training?
Before the Assessment
Again, assessments are not punishments or big scary exams. This is probably one of the most important things to communicate with everyone before going in. They’re just another training tool.
- Communicate about assessments in advance
- Use multiple channels to send reminders about upcoming assessments
- When an assessment is available, send an email/slack channel message/etc. in case the platform invite was missed
- Ensure everyone understands what's expected and what the completion dates are
- Promote assessments with things like posters and email graphics like you would for a tournament
After the Assessment
Once a round of assessments is complete, use what you’ve learned from the results to make the next ones even better. Take note of any patterns or common weaknesses across teams.
Consider scheduling time after assessments for teams to talk about what’s working and what they feel they need to improve on. Use this first-hand information to tailor training that addresses their concerns or requests, then create an assessment based on it.
This is where you can really help developers focus on things they care about and give them opportunities to upskill.
- Host a lunch and learn
- Focus on common missed practices within the industry as a whole
- Leverage the team’s assessment metrics
- Send out a survey - how did you find the assessment? Was it too long? Was it relevant to your day to day job activities?
- Provide detailed communication regarding the next steps and training
- Offer support and direction with Security Champions