Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Code completion for DABs includes #819

Closed
wants to merge 6 commits into from
Closed

Code completion for DABs includes #819

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
23a501e
Add support for test shards
fjakobs Jul 21, 2023
1897524
Add DABs code completion for includes
fjakobs Jul 25, 2023
010e484
Merge branch 'main' into dabs-include
fjakobs Jul 25, 2023
31d2793
Merge branch 'main' into dabs-include
fjakobs Jul 28, 2023
3d733ca
Address PR feedback
fjakobs Jul 28, 2023
c441042
Merge branch 'main' of github.com:databricks/databricks-vscode into d…
kartikgupta-db Sep 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion packages/databricks-vscode/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -688,7 +688,8 @@
"ansi-to-html": "^0.7.2",
"bcryptjs": "^2.4.3",
"triple-beam": "^1.4.1",
"winston": "^3.10.0"
"winston": "^3.10.0",
"yaml": "^2.3.1"
},
"devDependencies": {
"@istanbuljs/nyc-config-typescript": "^1.0.2",
Expand Down
69 changes: 56 additions & 13 deletions packages/databricks-vscode/src/bundle/GenerateBundle.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,64 @@
import {CliWrapper} from "../cli/CliWrapper";
import {extensions, Uri} from "vscode";
import path from "node:path";
import {workspace, RelativePattern, GlobPattern, Disposable} from "vscode";
import YAML from "yaml";

export async function generateBundleSchema(cli: CliWrapper) {
export async function generateBundleSchema(
cli: CliWrapper
): Promise<Disposable> {
// get freshly generated bundle schema
const bundleSchema = await cli.getBundleSchema();

// URI scheme for DABs JSON schemas
const dabsUriScheme = "dabs";

// URI for bundle root config json schema
const rootConfigSchemaUri = `${dabsUriScheme}:///root.json`;
const rootConfigSchemaUri = `${dabsUriScheme}:///dabs.json`;

const folder = workspace.workspaceFolders?.[0];
const configFilePattern = new RelativePattern(
folder!,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Return if no folder. Failing gracefully is better.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aren't we guaranteed to always have a workspace folder?

https://github.com/databricks/databricks-vscode/blob/main/packages/databricks-vscode/src/extension.ts#L71-L84

Still, I agree that we should make this code more defensive.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep. Just do not like the look of !

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I'll change it

"{databricks,bundle}.{yml,yaml}"
);

// file watcher on all YAML files
const watcher = workspace.createFileSystemWatcher("**/*.{yml,yaml}");
watcher.onDidChange(async () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use onDidCreate and onDidDelete

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also need a watcher on root bundle.yaml, so that we can update globs when the root is updated.
IMO we ONLY need the root watcher.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need an onDidCreate watcher for *.yml and an onDidChange watcher for {databricks|bundle}.{yaml|yml}.

That would be a bit more efficient but I feel that this code is easier to read.

We need the watcher on zonDidCreate in order to cover include patterns like *.yml.

await updateFileGlobs();
});

let configFiles = new Set<string>();
await updateFileGlobs();

async function updateFileGlobs() {
const fileGlobs: GlobPattern[] = [configFilePattern];

// find all YAML files that are included in the root config file
for (const configFile of await workspace.findFiles(configFilePattern)) {
try {
const fileContents = await workspace.fs.readFile(configFile);
const config = YAML.parse(fileContents.toString());
if (config.include) {
fileGlobs.push(
...config.include.map(
(g: string) => new RelativePattern(folder!, g)
)
);
}
} catch (e) {
// ignore errors
}
}

// expand globs to find all config files
const newConfigFiles = new Set<string>();
for (const glob of fileGlobs) {
for (const file of await workspace.findFiles(glob)) {
newConfigFiles.add(file.path);
}
}
configFiles = newConfigFiles;
}

const extensionYaml = extensions.getExtension("redhat.vscode-yaml");
if (extensionYaml) {
Expand All @@ -21,16 +69,9 @@ export async function generateBundleSchema(cli: CliWrapper) {
redHatYamlSchemaApi.registerContributor(
"dabs",
(resource: string) => {
const validFileNames: string[] = [
"databricks.yml",
"databricks.yaml",
"bundle.yml",
"bundle.yaml",
];
for (const name of validFileNames) {
if (path.basename(resource) === name) {
return rootConfigSchemaUri;
}
const resourceUri = Uri.parse(resource);
if (configFiles.has(resourceUri.path)) {
return rootConfigSchemaUri;
Comment on lines +89 to +91
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively, why can't we just call the updateFileGlobs here? It is not a very expensive function (for reasonable bundles) and less compute intensive than watchers.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code is executed whenever a file is opened. I'd like to keep that code path as fast as possible.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This need not be too expensive. All we need to do is match the resourceUri with all the include paths in the root bundle.yaml.

That is mostly string matching and regex matching for globs. We shouldn't even need filesystem access if we have a single watcher on root bundle.yaml keeping the list of includes updated.

}
return undefined;
},
Expand All @@ -43,4 +84,6 @@ export async function generateBundleSchema(cli: CliWrapper) {
}
);
}

return watcher;
}
6 changes: 4 additions & 2 deletions packages/databricks-vscode/src/extension.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -537,12 +537,14 @@ export async function activate(

// generate a json schema for bundle root and load a custom provider into
// redhat.vscode-yaml extension to validate bundle config files with this schema
generateBundleSchema(cli).catch((e) => {
try {
context.subscriptions.push(await generateBundleSchema(cli));
} catch (e) {
NamedLogger.getOrCreate("Extension").error(
"Failed to load bundle schema: ",
e
);
});
}

connectionManager.login(false).catch((e) => {
NamedLogger.getOrCreate(Loggers.Extension).error("Login error", e);
Expand Down
8 changes: 8 additions & 0 deletions yarn.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3715,6 +3715,7 @@ __metadata:
wdio-video-reporter: ^4.0.3
wdio-vscode-service: ^5.2.0
winston: ^3.10.0
yaml: ^2.3.1
yargs: ^17.7.2
languageName: unknown
linkType: soft
Expand Down Expand Up @@ -11198,6 +11199,13 @@ __metadata:
languageName: node
linkType: hard

"yaml@npm:^2.3.1":
version: 2.3.1
resolution: "yaml@npm:2.3.1"
checksum: 2c7bc9a7cd4c9f40d3b0b0a98e370781b68b8b7c4515720869aced2b00d92f5da1762b4ffa947f9e795d6cd6b19f410bd4d15fdd38aca7bd96df59bd9486fb54
languageName: node
linkType: hard

"yargs-parser@npm:20.2.4":
version: 20.2.4
resolution: "yargs-parser@npm:20.2.4"
Expand Down
Loading