Updated API review tokens to include two additional properties (#8885)

* Updated API review tokens to include two additional properties

tools/apiview/parsers/CONTRIBUTING.md

@@ -13,14 +13,39 @@ Specifically how to create or update a language parser to produce a hierarchy of
 
 ## Key concepts
 
-APIView token schema is available in [TypeSpec](./apiview-treestyle-parser-schema/model.tsp) and [JSON](./apiview-treestyle-parser-schema/model.json). Language parsers needs to create a `CodeFile` object as per the schema.
-`CodeFile` object contains metadata about the package and an array of `ReviewLine` objects. Each object of `ReviewLine` in `CodeFile` object is a top-level line to be listed. For e.g. Top-level lines are mostly namespace or module level nodes.
-Each `ReviewLine` object has children of `ReviewLine` to include sub nodes that needs to be listed in a review.
+APIView token schema is available in [TypeSpec](./apiview-treestyle-parser-schema/main.tsp) and [JSON](./apiview-treestyle-parser-schema/CodeFile.json). APIView requires all languages parser to create a JSON file as per the above mentioned schema to render the API review correctly. Navigation panel is automatically created based on the information in the tokens. Following are the main objects in the JSON schema.
 
-A sample token file is present [here](./apiview-treestyle-parser-schema/sample/Azure.Template_token.json).
+- CodeFile
+- ReviewLine
+- ReviewToken
+- CodeDiagnostic
 
-APIView generates a navigation tree based on the information in the tokens. A token is included in the navigation tree if `NavigatonDisplayName` is set in `ReviewToken` and `LineId` is set in `ReviewLine` object that contains the `ReviewToken`
+### CodeFile
+
+The `CodeFile` object represents the entire API review, which is rendered on the APIView tool. So the goal of language parser is to create a `CodeFile` object as per the [TypeSpec schema](./apiview-treestyle-parser-schema/main.tsp).
+
+`CodeFile` object contains an array of `ReviewLine` object, an array of `CodeDiagnostic` and some metadata about the package for e.g. package name, package version, language etc.  
+
+### ReviewLine
+
+`ReviewLine` object corresponds a line displayed on APIView and it contains the API review tree context.
+Each `ReviewLine` contains a list of `ReviewToken` which represents contents on the line.
+
+Each `ReviewLine` object also has children of `ReviewLine` to include sub nodes that needs to be listed in a review as sub-lines under current API review line.
+
+If a review line is related to another line at same level then id of related line should be set as `RelatedToLine` to ensure current line is not visible when related line is hidden. This use case is possible in diff view in which only modified node is visible.
+
+### ReviewToken
 
+A review line contains a list of `ReviewToken` and each `ReviewToken` represents a keyword, punctuation, text, type name etc. `ReviewToken` object also has a property that controls whether a space is required on review page after the current content. If the underlying content should act as an HREF then parser should set `NavigateToId` to the line ID of the target review line. `ReviewToken` can also be marked to be skipped from diff calculations when comparing two revisions by setting `SkipDiff` property of review token to true. For more information, refer the schema of `ReviewToken` in [TypeSpec schema](./apiview-treestyle-parser-schema/main.tsp)
+
+### CodeDiagnostic
+
+Code file contains a list of diagnostics detected by parser. Each `CodeDiagnostic` has a text, level and target review line ID. This also has optional diagnostic ID and help link URL.
+
+## How to include a line in navigation panel
+
+APIView generates a navigation tree based on the information in the tokens. A token is included in the navigation tree if `NavigatonDisplayName` is set in `ReviewToken` and `LineId` is set in `ReviewLine` object that contains the `ReviewToken`
 
 ## Serialization
 
@@ -31,15 +56,49 @@ Don't worry about indentation as it will be handled by the tree structure based
 
 A sample token file for .NET package `Azure.Template` is present [here](./apiview-treestyle-parser-schema/sample/Azure.Template_token.json) and corresponding rendered text representation is available [here](./apiview-treestyle-parser-schema/sample/Azure.Template_review_content.txt).
 
+See following sample API review line and corresponding `ReviewLine` token.
+
+API Review Line: `namespace Azure.Data.Tables {`
+
+```json
+{
+  "LineId": "Azure.Data.Tables",
+  "Tokens": [
+    {
+      "Kind": 2,
+      "Value": "namespace",
+      "HasSuffixSpace": true,
+      "RenderClasses": []
+    },
+    {
+       "Kind": 0,
+       "Value": "Azure.Data.Tables",
+       "NavigationDisplayName":"Azure.Data.Tables",
+       "HasSuffixSpace": true,
+       "RenderClasses": [
+          "namespace"
+        ]
+    },
+    {
+       "Kind": 1,
+       "Value": "{",
+       "HasSuffixSpace": true,
+       "RenderClasses": []
+    }
+  ],
+  "Children": [],
+  "IsHidden": false
+}
+```
+
 
 ## JSON token validation
 
 You can validate JSON tokens against required JSON schema using [JSON schema validator](https://www.jsonschemavalidator.net/).
 
-- Select `Custom` as schema type and copy and paste the contents in [json schema](./apiview-treestyle-parser-schema/model.json) to left panel.
+- Select `Custom` as schema type and copy and paste the contents in [json schema](./apiview-treestyle-parser-schema/CodeFile.json) to left panel.
 - Generate APIView token file and paste generated JSON content onto right side panel to validate.
 
-
 ## Get help
 
 Please reach out at [APIView Teams Channel](https://teams.microsoft.com/l/channel/19%3A3adeba4aa1164f1c889e148b1b3e3ddd%40thread.skype/APIView?groupId=3e17dcb0-4257-4a30-b843-77f47f1d4121&tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47) if you need more information.

tools/apiview/parsers/apiview-treestyle-parser-schema/.gitignore

@@ -0,0 +1,9 @@
+# MacOS
+.DS_Store
+
+# Default TypeSpec output
+tsp-output/
+dist/
+
+# Dependency directories
+node_modules/

tools/apiview/parsers/apiview-treestyle-parser-schema/model.json → tools/apiview/parsers/apiview-treestyle-parser-schema/CodeFile.json

@@ -131,7 +131,15 @@
                 },
                 "IsHidden": {
                     "type": "boolean",
-                    "description": "Set current line as hidden code line by default. .NET has hidden APIs and architects does not want to see them by default."
+                    "description": "Set current line as hidden code line by default. .NET has hidden APIs and architects don't want to see them by default."
+                },
+                "IsContextEndLine": {
+                    "type": "boolean",
+                    "description": "Set current line as context end line. For e.g. line with token } or empty line after the class to mark end of context."
+                },
+                "RelatedToLine": {
+                    "type": "string",
+                    "description": "Set ID of related line to ensure current line is not visible when a related line is hidden.\nOne e.g. is a code line for class attribute should set class line's Line ID as related line ID."
                 }
             },
             "required": [
@@ -143,14 +151,16 @@
             "type": "object",
             "properties": {
                 "DiagnosticId": {
-                    "type": "string"
+                    "type": "string",
+                    "description": "Diagnostic ID is auto generated ID by CSharp analyzer."
                 },
                 "TargetId": {
                     "type": "string",
-                    "description": "Id of ReviewLine object where this diagnostic needs to be displayed(Options"
+                    "description": "Id of ReviewLine object where this diagnostic needs to be displayed"
                 },
                 "Text": {
-                    "type": "string"
+                    "type": "string",
+                    "description": "Auto generated system comment to be displayed under targeted line."
                 },
                 "Level": {
                     "$ref": "#/$defs/CodeDiagnosticLevel"
@@ -177,7 +187,7 @@
                 },
                 "NavigationDisplayName": {
                     "type": "string",
-                    "description": "NavigationDisplayName can be used set navigation dispaly name to be used in navigation tree.\nThis value will be used in the navigation if NavigationDisplayName is not set."
+                    "description": "NavigationDisplayName is used to create a tree node in the navigation panel. Navigation nodes will be created only if token contains navigation display name."
                 },
                 "NavigateToId": {
                     "type": "string",

tools/apiview/parsers/apiview-treestyle-parser-schema/model.tsp → tools/apiview/parsers/apiview-treestyle-parser-schema/main.tsp

@@ -1,9 +1,3 @@
-import "@typespec/json-schema";
-
-using TypeSpec.JsonSchema;
-
-@jsonSchema
-
 /** ReviewFile represents entire API review object. This will be processed to render review lines. */
 model CodeFile {
   PackageName: string;
@@ -36,6 +30,12 @@ model ReviewLine {
   Children?: Array<ReviewLine>;
   /** Set current line as hidden code line by default. .NET has hidden APIs and architects don't want to see them by default. */
   IsHidden?: boolean;
+  /** Set current line as context end line. For e.g. line with token } or empty line after the class to mark end of context. */
+  IsContextEndLine?: boolean;
+  /** Set ID of related line to ensure current line is not visible when a related line is hidden.
+   * One e.g. is a code line for class attribute should set class line's Line ID as related line ID.
+  */
+  RelatedToLine?: string;
 }