Documentation, assistant-base, and python example updates (#27)

* add git config autoSetupRemote to devcontainer setup

* improved handling of config from env

* better azure app service logging support

* async keyvault api

* adds recommended vscode extensions

* improved documentation

* additional clean up for python example

.devcontainer/README.md

@@ -1,23 +1,64 @@
 # Using GitHub Codespaces with devcontainers for Semantic Workbench
 
-- Create app registration at https://portal.azure.com
-  - Name: Semantic Workbench
-  - Supported account types: Accounts in any organizational directory and personal Microsoft accounts
-  - Redirect URI: Single-page application (SPA) & `https://<YOUR_CODESPACE_HOST>-4000.app.github.dev`
-  - Copy the Application (client) ID from Overview
-- Update `constants.ts` with the Application (client) ID
-- Update `middleware.py` with the Application (client) ID
-- Use VS Code > Run and Debug > `Semantic Workbench` to start both the app and the service
-- After launching semantic-workbench-service, go to Ports and make 3000 public
-- In the VS Code terminal tab, find the semantic-workbench-app and click on the link to open the app
+This folder contains the configuration files for using GitHub Codespaces with devcontainers for Semantic Workbench and assistant development.
 
-## Python assistant example
+GitHub Codespaces is a feature of GitHub that provides a cloud-based development environment for your repository. It allows you to develop, build, and test your code in a consistent environment, without needing to install dependencies or configure your local machine.
 
-We have included an example Python assistant that echos the user's input and can serve as a starting point for your own assistant.
+## Why
 
-See the [Python assistant example README](../examples/python-example01/README.md) for more details.
+- **Consistent environment**: All developers use the same environment, regardless of their local setup.
+- **Platform agnostic**: Works on any system with a web browser and internet connection, including Chromebooks, tablets, and mobile devices.
+- **Isolated environment**: The devcontainer is isolated from the host machine, so you can install dependencies without affecting your local setup.
+- **Quick setup**: You can start developing in a few minutes, without needing to install dependencies or configure your environment.
 
-## TODO
+## How to use
 
-- [ ] Add support for reading Application (client) ID from environment variables
-- [ ] Improve this README with details on App Registration setup, and more detailed steps
+### Pre-requisites
+
+#### Create a new GitHub Codespace
+
+- Open the repository in GitHub Codespaces
+  - Navigate to the [repository in GitHub](https://github.com/microsoft/semanticworkbench)
+  - Click on the `Code` button and select the `Codespaces` tab
+  - Click on the `Codespaces` > `+` button
+  - Allow the Codespace to build and start, which may take a few minutes - you may continue to the next steps while it builds
+  - Make a note of your Codespace host name, which is the part of the URL before `.github.dev`.
+
+#### Set up the workbench app and service
+
+While the Codespaces environment makes it easy to start developing, the hard-coded app registration details in the app and service cannot support a wildcard redirect URI. This means you will need to create your own Azure app registration and update the app and service files with the new app registration details.
+
+Follow the instructions in the [Custom app registration](../docs/CUSTOM_APP_REGISTRATION.md) guide to create a new Azure app registration and update the app and service files with the new app registration details.
+
+### Using the Codespace in VS Code
+
+#### Launch the Codespace
+
+- Open the repository in VS Code
+  - Click on the `Code` button and select the Codespaces tab and choose the Codespace you created
+  - Since the Codespace is available as a Progressive Web App (PWA), you can also run the Codespace as a local app (in its own window, taskbar/dock/home icon, etc.) for quicker access. See the following links for general information on installing PWAs:
+    - [Microsoft Edge](https://learn.microsoft.com/en-us/microsoft-edge/progressive-web-apps-chromium/ux)
+    - Search the internet for `install PWA on <browser/platform>` for help with other browsers/platforms
+
+#### Start the app and service
+
+- Use VS Code > `Run and Debug` (Ctrl/Cmd+Shift+D) > `Semantic Workbench` to start both the app and the service
+- Once `semantic-workbench-service` has launched, you will need to make it `public`
+  - The app uses the service to fetch data and it must be accessible from the browser of the user logged into the app, which is why the service endpoint must be public. This also means it is accessible to anyone with the URL. To provide a minimal level of security, the app registration is required. Any calls to the service must include an access token from a user logged into the configured app registration.
+  - In VS Code go to the `ports` tab (same pane as the `terminal` where the services are running)
+  - Right-click on the `service:semantic-workbench-service` (Port 3000) and select `Port Visibility` > `Public`
+- In the VS Code `terminal` tab, find the `semantic-workbench-app` and click on the `Local` link to open the app
+  - This will automatically open the app in a new browser tab, navigating to the correct Codespace host and setting the necessary header values to privately access the app
+- You can now interact with the app and service in the browser
+- Next steps:
+  - Launch an assistant service, using an [example assistant](../examples/) or your own assistant
+    - If launching an assistant service from within the same Codespace, it will be automatically accessible to the Semantic Workbench service
+  - Add the assistant to the workbench app by clicking the `Add Assistant` button in the app and selecting the assistant from the list
+  - Configure the assistant and interact with it in the app by clicking on the assistant in the list
+  - From the assistant configuration screen, click `New Conversation` to start a new conversation with the assistant
+
+## Assistant service example
+
+We have included an example Python assistant service that echos the user's input and can serve as a starting point for your own assistant service.
+
+See the [python-example01/README](../examples/python-example01/README.md) for more details.

.devcontainer/devcontainer.json

@@ -44,7 +44,9 @@
     // Build and restore dependencies for key projects in the repo
     "make_workbench_app": "make -C /workspaces/semanticworkbench/semantic-workbench/v1/app",
     "make_workbench_service": "make -C /workspaces/semanticworkbench/semantic-workbench/v1/service",
-    "make_python_examples01": "make -C /workspaces/semanticworkbench/examples/python-examples01"
+    "make_python_examples01": "make -C /workspaces/semanticworkbench/examples/python-example01",
+    // Set up git to automatically set up the remote when pushing if it doesn't exist
+    "git_config": "git config --add push.autoSetupRemote true"
   },
 
   // Configure tool-specific properties.
@@ -58,7 +60,8 @@
         "esbenp.prettier-vscode",
         "ms-python.black-formatter",
         "ms-python.flake8",
-        "ms-python.python"
+        "ms-python.python",
+        "streetsidesoftware.code-spell-checker"
       ]
     }
   }

.vscode/extensions.json

@@ -0,0 +1,11 @@
+{
+  "recommendations": [
+    "aaron-bond.better-comments",
+    "dbaeumer.vscode-eslint",
+    "esbenp.prettier-vscode",
+    "ms-python.black-formatter",
+    "ms-python.flake8",
+    "ms-python.python",
+    "streetsidesoftware.code-spell-checker"
+  ]
+}

.vscode/settings.json

@@ -153,5 +153,40 @@
       "bold": false,
       "italic": false
     }
+  ],
+  "cSpell.words": [
+    "aarontamasfe",
+    "appsettings",
+    "arcname",
+    "asgi",
+    "azurewebsites",
+    "cachetools",
+    "Codespace",
+    "Codespaces",
+    "dbaeumer",
+    "devcontainer",
+    "devcontainers",
+    "devtunnel",
+    "dotenv",
+    "esbenp",
+    "fastapi",
+    "hashkey",
+    "httpx",
+    "jsonlogger",
+    "jungaretti",
+    "keyvault",
+    "levelname",
+    "levelno",
+    "msal",
+    "pydantic",
+    "pylance",
+    "pyproject",
+    "pythonjsonlogger",
+    "semanticworkbench",
+    "sqlalchemy",
+    "sqlmodel",
+    "tracebacks",
+    "webservice",
+    "winget"
   ]
 }

README.md

@@ -21,7 +21,13 @@ in various development environments.
 
 ![Semantic Workbench architecture](docs/architecture-animation.gif)
 
-# Quick start
+# Quick start (Recommended) - GitHub Codespaces for turn-key development environment
+
+GitHub Codespaces provides a cloud-based development environment for your repository. It allows you to develop, build, and test your code in a consistent environment, without needing to install dependencies or configure your local machine. It works with any system with a web browser and internet connection, including Windows, MacOS, Linux, Chromebooks, tablets, and mobile devices.
+
+See the [GitHub Codespaces / devcontainer README](.devcontainer/README.md) for more information on how to set up and use GitHub Codespaces with Semantic Workbench.
+
+# Quick start - Local development environment
 
 - Start the backend service, see [here for instructions](semantic-workbench/v1/service/README.md).
 - Start the frontend app, see [here for instructions](semantic-workbench/v1/app/README.md).

docs/ASSISTANT_DEVELOPMENT_GUIDE.md

@@ -3,15 +3,22 @@
 ## Overview
 
 For assistants to be instantiated in Semantic Workbench, you need to implement an assistant service that the workbench can talk with via http.
+
 We provide several python base classes to make this easier: [semantic-workbench-assistant](../semantic-workbench/v1/service/semantic-workbench-assistant/README.md)
 
 Example assistant services:
 
 - [Canonical assistant example](../semantic-workbench/v1/service/semantic-workbench-assistant/semantic_workbench_assistant/canonical.py)
+- Python assistant [example 1](../examples/python-example01/README.md)
 - .NET agent [example 1](../examples/dotnet-example01/README.md) and [example 2](../examples/dotnet-example02/README.md)
 
 ## Top level concepts
 
+RECOMMENDED FOR PYTHON: Use the `semantic-workbench-assistant` base classes to create your assistant service. These classes provide a lot of the boilerplate code for you.
+
+See the [semantic-workbench-assistant.assistant_base](../service/semantic-workbench-assistant/semantic_workbench_assistant/assistant_base.py) for the base classes
+and the Python [example 1](../examples/python-example01/README.md) for an example of how to use them.
+
 ### assistant_service.FastAPIAssistantService
 
 Your main job is to implement a service that supports all the Semantic Workbench methods. The [Canonical assistant example](../semantic-workbench/v1/service/semantic-workbench-assistant/semantic_workbench_assistant/canonical.py) demonstrates a minimal implementation.
@@ -22,42 +29,36 @@ It implements an assistant service that inherits from FastAPIAssistantService:
 
 This service is now a FastAPIAssistantService containing all the assistant methods that can be overridden as desired.
 
-## Node Engine assistants
-
-Though not mandatory, you might find it helpful to use the [Node Engine](https://github.com/microsoft/nodeengine) for creating assistants.
-Node Engine gives you utilities for defining complex LLM interactions using a flow-based architecture where a "context" is modified through several flow components.
-We have provided a NodeEngineAssistant base class that makes using the Node Engine simpler.
+## Assistant service development: general steps
 
-### Inheriting BaseNodeEngineAssistantInstance in your AssistantInstanceModel
+- Set up your dev environment
+  - SUGGESTED: Use GitHub Codespaces for a quick, easy, and consistent dev
+    environment: [/.devcontainer/README.md](../.devcontainer/README.md)
+  - ALTERNATIVE: Local setup following the [main README](../README.md#quick-start---local-development-environment)
+- Create a dir for your projects. If you create this in the repo root, any assistant example projects will already have the correct relative paths set up to access the `semantic-workbench-*` packages or libraries.
+- Create a project for your new assistant in your projects dir, e.g. `/<your_projects>/<your_assistant_name>`
+- Getting started with the assistant service
+  - Copy skeleton of an existing project: e.g. one of the projects from the [examples](../examples) directory
+    - Alternatively, consider using the canonical assistant as a starting point if you want to implement a new custom base
+  - Set up .env
+- Build and Launch assistant. Run workbench service. Run workbench app. Add assistant local url to workbench via UI.
+- NOTE: See additional documentation in [/semantic-workbench/v1/app/docs](../semantic-workbench/v1/app/docs/) regarding app features that can be used in the assistant service.
 
-You need to create an instance model for your assistant, e.g. `AssistantInstanceModel`.
-It should implement the `node_engine_assistant.BaseAssistantInstance` Pydantic model and ABC.
-This Pydantic model holds the id, assistant_name, config, and conversations attributes and the ABC ensures that my assistant sets up `config`, implements a `construct` factory static method, and a `config_ui_schema` static method.
-These static methods are used by the workbench service.
+## Assistant service deployment
 
-Your assistant (service) should inherit `BaseNodeEngineAssistant` typed as my `AssistantInstanceModel`.
-For example:
+DISCLAIMER: The security considerations of hosting a service with a public endpoint are beyond the scope of this document. Please ensure you understand the implications of hosting a service before doing so. It is not recommended to host a publicly available instance of the Semantic Workbench app.
 
-```jsx
-MyAssistant(
-  node_engine_assistant.BaseNodeEngineAssistant[AssistantInstanceModel]
-):
-```
+If you want to deploy your assistant service to a public endpoint, you will need to create your own Azure app registration and update the app and service files with the new app registration details. See the [Custom app registration](../docs/CUSTOM_APP_REGISTRATION.md) guide for more information.
 
-`BaseNodeEngineAssistant` is an ABC generic for any type that inherits from `BaseAssistantInstance` (like my `AssistantInstanceModel`).
-This class inherits `assistant_service.FastAPIAssistantService` which is an ABC for all the service methods.
-`assistant_service_api(app: FastAPI, service: FastAPIAssistantService)` is a function that configures the app as a `FastApiAssistantService` (implementing its required ABC methods), the implemented methods forwards method calls to the provided `FastAPIAssistantService`.
+### Steps
 
-## Assistant service development: general steps
+TODO: Add more detailed steps, this is a high-level overview
 
-- Set up your dev environment
-  - Create a dir for your stuff
-  - Create a project there
-  - copy dev env skeleton files: .black.toml, Dockerfile, .gitignore, .flake8, Makefile, pyproject.toml which has all of the above files, ready to go.
-  - Update README.md
-- Getting started with the assistant service
-  - copy skeleton of project: e.g. a node engine assistant
-  - Consider using the canonical assistant as a starting point,
-  - or... copy node_engine scaffolding (make sure to create **init**.py files and a service.py file)
-  - Set up .env
-- Build and Launch assistant. Run workbench service. Run workbench app. Add assistant local url to workbench via UI.
+- Create a new Azure app registration
+- Update your app and service files with the new app registration details
+- Deploy your service to a public endpoint
+- Update the workbench app with the new service URL
+- Deploy your assistant service to an endpoint that the workbench service can access
+  - Suggested: limit access to the assistant service to only the workbench service using a private network or other security measures)
+- Deploy the workbench app to an endpoint that users can access
+  - Suggested: limit access to the workbench app to only trusted users

docs/CUSTOM_APP_REGISTRATION.md

@@ -0,0 +1,39 @@
+# Custom app registration
+
+The code in this repo is intended to allow for quick-to-try usage. This includes a hard-coded app registration details in the app and service. While this works for minimal setup for development and usage in localhost environments, you will need to create your own Azure app registration and update the app and service files with the new app registration details if you want to use this in a hosted environment.
+
+**DISCLAIMER**: The security considerations of hosting a service with a public endpoint are beyond the scope of this document. Please ensure you understand the implications of hosting a service before doing so. It is **not recommended** to host a publicly available instance of the Semantic Workbench app.
+
+## Create a new Azure app registration
+
+### Prerequisites
+
+In order to complete these steps, you will need to have an Azure account. If you don't have an Azure account, you can create a free account by navigating to https://azure.microsoft.com/en-us/free.
+
+App registration is a free service, but you may need to provide a credit card for verification purposes.
+
+### Steps
+
+- Navigate to the [Azure portal > App registrations](https://portal.azure.com/#view/Microsoft_AAD_RegisteredApps/ApplicationsListBlade)
+  - Click on `New registration` and fill in the details
+  - Name: `Semantic Workbench` (or any name you prefer)
+  - Supported account types: `Accounts in any organizational directory and personal Microsoft accounts`
+  - Redirect URI: `Single-page application (SPA)` & `https://<YOUR_HOST>`
+    - Example (if using [Codespaces](../.devcontainer/README.md)): `https://<YOUR_CODESPACE_HOST>-4000.app.github.dev`
+  - Click on `Register`
+- View the `Overview` page for the newly registered app and copy the `Application (client) ID` for the next steps
+
+## Update your app and service files with the new app registration details
+
+Edit the following files with the new app registration details:
+
+- Semantic Workbench app: [constants.ts](../semantic-workbench/v1/app/src/Constants.ts)
+
+  - Update the `msal.auth.clientId` with the `Application (client) ID`
+
+- Semantic Workbench service: [middleware.py](../semantic-workbench/v1/service/semantic-workbench-service/semantic_workbench_service/middleware.py)
+  - Update the `allowed_app_ids` with the `Application (client) ID`
+
+## TODO
+
+- [ ] Update the codebase to allow app registration details to be passed in as environment variables

docs/LOCAL_ASSISTANT_WITH_REMOTE_WORKBENCH.md

@@ -1,6 +1,7 @@
 # Local Assistant / Remote Semantic Workbench
 
 This guide will walk you through the process of creating a local assistant that communicates with a remote Semantic Workbench instance.
+
 This guide assumes you have already set up a Semantic Workbench instance and have a basic understanding of how to create an assistant.
 
 ## Prerequisites
@@ -11,12 +12,12 @@ This guide assumes you have already set up a Semantic Workbench instance and hav
 ## Steps
 
 [Microsoft Dev Tunnel](https://learn.microsoft.com/en-us/azure/developer/dev-tunnels/) is a tool that allows you to securely expose your local development environment to the cloud. This is useful for testing your local assistant with a remote Semantic Workbench instance.
+
 The following steps will guide you through the process of setting up a tunnel to your local machine.
 
 **Note:** The tunnel will only be accessible while it is running, so you will need to keep the tunnel running while testing and may not want to share the URL with others without being explicit about the availability.
 
-**SECURITY NOTE:** Be aware that the tunnel will expose your local machine to the internet and allow anonymous access to the assistant.
-Make sure you are aware of the security implications and take appropriate precautions.
+**SECURITY NOTE:** Be aware that the tunnel will expose your local machine to the internet and allow anonymous access to the assistant. Make sure you are aware of the security implications and take appropriate precautions.
 
 - [Install](https://learn.microsoft.com/en-us/azure/developer/dev-tunnels/get-started?tabs=windows#install)
   - `winget install Microsoft.devtunnel` (PowerShell)