Conversation
in4margaret
force-pushed
the
mostap/githubActions
branch
from
220d6ca to
240a9fe
Compare
| - [Dialogflow endpoint configuration](docs/DialogflowEndpointConfiguration.md) | ||
| - [Configuring LUIS CI/CD with Azure Pipelines](docs/AzurePipelines.md) | ||
| - [Extending the CLI to new NLU providers](docs/CliExtensions.md) | ||
| - [GitHub Actions workflow using NLU.DevOps cli tool](docs/NLUGitHubActions.md) |
There was a problem hiding this comment.
For consistency, rename to Configuring LUIS CI/CD with GitHub Actions
| This document covers setting up a CI pipeline for LUIS using GitHub Actions and NLU.DevOps. | ||
| The workflow will be the following: | ||
| 1. create, train, publish the LUIS model using sample utterances | ||
| 2. send a test set to created LUIS model |
There was a problem hiding this comment.
run a batch test against the model through the LUIS prediction API
| The workflow will be the following: | ||
| 1. create, train, publish the LUIS model using sample utterances | ||
| 2. send a test set to created LUIS model | ||
| 3. evaluate model by comparing results received from LUIS with expected values |
There was a problem hiding this comment.
analyze the F<sub>1</sub> scores for intents and entities in the model
| 1. create, train, publish the LUIS model using sample utterances | ||
| 2. send a test set to created LUIS model | ||
| 3. evaluate model by comparing results received from LUIS with expected values | ||
| 4. delete the model from the portal |
There was a problem hiding this comment.
delete the LUIS app resources created for testing
| 3. evaluate model by comparing results received from LUIS with expected values | ||
| 4. delete the model from the portal | ||
|
|
||
| Supply a name for the action and set it up to trigger on pull requests. |
There was a problem hiding this comment.
Seems like we should start a new sub-section here.
|
|
||
| ``` | ||
| - name: Train Luis model | ||
| run: dotnet nlu train -s luis -u utterances.json --save-appsettings |
There was a problem hiding this comment.
This probably won't work without also including the -m settings.luis.json
| run: echo "::add-path::$HOME/.dotnet/tools" | ||
| ``` | ||
| We use [utterances.json](../models/utterances.json) for training. You can replace this file with another file that consists of intents, utterances, entities that you need for your own model. | ||
| More about the format of this file [here](https://github.com/microsoft/NLU.DevOps/blob/master/docs/GenericUtterances.md). |
There was a problem hiding this comment.
You can just use the relative path [here](GenericUtterances.md), which will make this work better for our GitHub pages.
| ``` | ||
| We use [utterances.json](../models/utterances.json) for training. You can replace this file with another file that consists of intents, utterances, entities that you need for your own model. | ||
| More about the format of this file [here](https://github.com/microsoft/NLU.DevOps/blob/master/docs/GenericUtterances.md). | ||
| To train your model we should add the following: |
There was a problem hiding this comment.
Clarify that we're using the NLU.DevOps CLI for training the LUIS model, but you can also use any other tool, such as the Bot Framework CLI.
| More about the format of this file [here](https://github.com/microsoft/NLU.DevOps/blob/master/docs/GenericUtterances.md). | ||
| To train your model we should add the following: | ||
|
|
||
| ``` |
There was a problem hiding this comment.
mark these code fences as yaml or json where applicable.
| luisAuthoringKey: ${{ secrets.luisAuthoringKey }} | ||
| ``` | ||
|
|
||
| More about the command [here](https://github.com/microsoft/NLU.DevOps/blob/master/docs/Train.md). |
| luisAuthoringKey: ${{ secrets.luisAuthoringKey }} | ||
| ``` | ||
|
|
||
| More about the command [here](https://github.com/microsoft/NLU.DevOps/blob/master/docs/Train.md). |
There was a problem hiding this comment.
... about the train command...
| luisAuthoringKey: ${{ secrets.luisAuthoringKey }} | ||
| ``` | ||
|
|
||
| You can find GitHub Action workflow yaml file [here](../pipelines/.github/workflows/nlugithub.yml). |
| on: [pull_request] | ||
| ``` | ||
|
|
||
| 1. Install NLU.DevOps CLI tool on GitHub agent. |
There was a problem hiding this comment.
Rather than using ordered list items, maybe we should just break these into sub-sections. I.e., this one could be called:
### Installing NLU.DevOps on the GitHub Actions agent
| ``` | ||
| We use [utterances.json](../models/utterances.json) for training. You can replace this file with another file that consists of intents, utterances, entities that you need for your own model. | ||
| More about the format of this file [here](https://github.com/microsoft/NLU.DevOps/blob/master/docs/GenericUtterances.md). | ||
| To train your model we should add the following: |
There was a problem hiding this comment.
Create sub-section header for training.
|
|
||
| Check out the [LUIS docs](https://docs.microsoft.com/en-us/azure/cognitive-services/luis/luis-how-to-azure-subscription) for more information on where to find your authoring or runtime keys. | ||
|
|
||
| 2. To test LUIS model let's use [utterancesTest.json](../models/utterancesTest.json) file. |
There was a problem hiding this comment.
Create sub-section header for testing
| 2. To test LUIS model let's use [utterancesTest.json](../models/utterancesTest.json) file. | ||
| We can save the result in results.json. During the training step, we may have created a new LUIS application. By using the `--save-appsettings` flag, the LUIS application ID is stored locally in a settings file that is picked up by subsequent NLU.DevOps CLI commands. | ||
|
|
||
| Yaml may look like that: |
| - name: Test Luis model | ||
| run: dotnet nlu test -s luis -u utterancesTest.json -o results.json | ||
| env: | ||
| luisAuthoringRegion: ${{ secrets.luisAuthoringRegion }} |
There was a problem hiding this comment.
We may want to discuss that NLU.DevOps will attempt to use your authoring key, but this is limited. Add details that you can also specify the luisEndpointKey and luisEndpointRegion if your app has an assigned prediction resource, and that you can also use the REST APIs to assign a prediction resource.
| luisAuthoringRegion: ${{ secrets.luisAuthoringRegion }} | ||
| luisAuthoringKey: ${{ secrets.luisAuthoringKey }} | ||
| ``` | ||
| 3. We use the `compare` command from the NLU.DevOps CLI to evaluate the results. The expected intents and entities in this case are given in the `utterancesTest.json` file, while the results predicted by the LUIS model are in the `results.json` file. |
There was a problem hiding this comment.
Section heading for comparing the model.
| ``` | ||
|
|
||
| If you open your GitHub workflow run step for this command in the console, you can see something similar to | ||
|  |
There was a problem hiding this comment.
Rather than a screenshot, can we inline the bash output in a code fence? It makes it easier to modify in case we make change the output expectations in the future.
Also, we could add the markdown table output for F1 scores as well.
| If you open your GitHub workflow run step for this command in the console, you can see something similar to | ||
|  | ||
|
|
||
| 4. When you work on several hypotheses, sometimes you need only to get results and you don't want to keep the model. It is possible to delete the model in the same pipeline after you get results. |
There was a problem hiding this comment.
Add section header for cleaning up LUIS app resources.
| luisAuthoringKey: ${{ secrets.luisAuthoringKey }} | ||
| ``` | ||
|
|
||
| You can find GitHub Action workflow yaml file [here](../pipelines/.github/workflows/nlugithub.yml). |
There was a problem hiding this comment.
Move this up to the top of the document and say something like "We have an end-to-end example of the action steps below here"
| "text": "what's the weather like", | ||
| "intent": "None" | ||
| } | ||
| ] |
There was a problem hiding this comment.
tabs are off on the closing ]. Also, additional whitespace on the last line.
| @@ -0,0 +1,50 @@ | |||
| name: CINLU | |||
There was a problem hiding this comment.
Don't worry about moving this to the top-level, we probably should not run the same tests twice.
| - name: Path | ||
| run: echo "::add-path::$HOME/.dotnet/tools" | ||
|
|
||
| - name: Train Luis model |
| with: | ||
| dotnet-version: 2.2.108 | ||
|
|
||
| - name: Install dotnet-nlu |
| luisAuthoringRegion: ${{ secrets.luisAuthoringRegion }} | ||
| luisAuthoringKey: ${{ secrets.luisAuthoringKey }} | ||
|
|
||
| - name: Test Luis model |
| luisAuthoringRegion: ${{ secrets.luisAuthoringRegion }} | ||
| luisAuthoringKey: ${{ secrets.luisAuthoringKey }} | ||
|
|
||
| - name: Compare Luis model |
There was a problem hiding this comment.
Compare Luis model -> Compare LUIS results
| - name: Compare Luis model | ||
| run: dotnet nlu compare -e utterancesTest.json -a results.json | ||
|
|
||
| - name: Archive TestResult |
There was a problem hiding this comment.
Since we're not doing anything with the results (downloading them), maybe just remove this for now.
| name: TestResult | ||
| path: TestResult.xml | ||
|
|
||
| - name: Delete Luis model |
| env: | ||
| luisAuthoringRegion: ${{ secrets.luisAuthoringRegion }} | ||
| luisAuthoringKey: ${{ secrets.luisAuthoringKey }} | ||
|
No newline at end of file |
There was a problem hiding this comment.
Looks like some extra whitespace at the end of the file.
3 participants
Resolves #244