I recently reviewed the two published drafts for the ISASecure Embedded Device Security Assurance Certification and had a number of comments on how easy or hard it would be for third party testing of the requirements. Since that review ISASecure was kind enough to send another version of the Functional Security Assessment and Software Design Security Assessment that included two additional columns for each requirement: Validation Activity and Validation by Independent Test Required (Yes/No).
While I can’t share the documents with our loyal readers, I am allowed to give you my thoughts on the full version of the documents. Today I’ll cover the validation portion of Functional Security Assessment document.
Functional Security Assessment Document
The thing that jumped out at me first is that validation is almost exclusively a documentation review. Virtually all of the validation activities begin with either “Verify user documents include evidence” … or “Verify design documents include evidence” … This is then followed by a restatement of the requirement, sometimes in more specificity and sometimes less.
After the restatement of the requirement, the Validation Activity has the possible test results listed, typically Supported, Allocated and Non-Supported. Now many of the requirements have no description for Supported and Non-Supported and a generic “able to be met by other components” for Allocated. There are some interesting Validation Activity results if you can get to Use Control requirements in the middle of the document. For example:
FSA-DI-1.5 – Basic Modification of Transmitted Data: The IACS embedded device shall employ basic mechanisms to recognize changes to information during communication independent of the basic communication protocol stack
Validation Activity:
a. Supported – via application level CRC of at least 16 bit length or duplicated data, or
b. Allocatable – via restriction of network confined to physically protected area or protected by another layer
independent of this device prior to leaving protected area
c. Not Supported – includes dependence on Link layer CRC for detection of corruption as this is insufficient against malicious intent
Without commenting on the appropriateness of the measure, a tester would at least have some guidance on what Supported meant. Another example is FSA-DI-1.1 – Insertion of Data Packets with “Supported – has a specific feature such as sequence numbering to detect insertions”. It may get interesting if a tester has to evaluate a solution other than sequence numbering, but this does provide some guidance to the certification group on what ISASecure was expecting.
However, with this limited level of specificity there will be challenges for a certification agency. For instance, how will different groups and people consistently
“Examine design documents and determine if there are security measures to provide verification of mission critical data conforms to expected formats and limits and record results”.
One could easily test that something is taking place, but how much is enough, what is mission critical data, etc. This just points out that additional audit guidance documentation will need to be developed, which is not a surprise.
About 90% of the requirements specify Independent Test, but some don’t such as Role Based Access Control [FSA-AC-1.1], Management of Passwords [FSA-AC-2.1.5], or Authentication Feedback [FSA-AC-2.5]. This is oddly inconsistent, and I couldn’t deduce any reason for the small number of “no independent testing” requirements. Since the “testing” is mostly document review it seems odd why any requirements that were worthy of being included in the document would be “no”.
Conclusions
These additional validation columns do provide some benefits in understanding how the Functional Security Assessment will be performed, but a lot more detail is needed to hand this testing over to third parties and have it consistently applied.
The Functional Security Assessment portion of the Embedded Device Security Assessment is primarily a paper exercise. This has some value, but I’m not sure why these features are not documented and then tested to verify the implementation meets the documentation. Currently the Levels envisioned for the specification only deal with the rigor of the requirements, but perhaps there should be some sort of level for the rigor of the assessment methodology as well. Even two levels of document review and document plus device testing would be one idea.
However the last thing ISASecure is derail or delay the efforts to tackle multiple levels of testing rigor. They need to get something out and a success under their belt as soon as possible.
Up next is the review of the validation measures in the Software Development Security Assessment.