Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add scanning of local files and folders (single scan) #558

Merged
merged 9 commits into from
Aug 7, 2024
35 changes: 27 additions & 8 deletions doc/quick-start-guide-and-build.md
Original file line number Diff line number Diff line change
Expand Up @@ -245,34 +245,53 @@ _LPVS_ is now built and running. You can create a new pull request or update an

#### 4.2 Single scan mode

Alternatively, you can perform a one-time scan on a specific pull request using the single scan mode. Follow these steps:
Alternatively, you can perform a one-time scan on a specific pull request or local files using the single scan mode. Follow these steps:

4.2.1 Begin by running the installation and navigating to the target directory, similar to the process in service mode (refer to steps 4.1.1 and 4.1.2):
4.2.1 Install and navigate to the target directory as described in step 4.1.1 and 4.1.2:

```bash
mvn clean install
cd target/
```

4.2.2 Execute the single scan with the following command:
4.2.2 Choose either to scan a specific pull request from GitHub or local files.

4.2.2.1 To scan a specific pull request from GitHub, execute the following command:

```bash
java -jar -Dgithub.token=<my-token> lpvs-*.jar --github.pull.request=<PR URL>
```

4.2.3 By default, the above command requires a pre-configured MySQL database. To avoid setting up the database, use the "singlescan" profile:
4.2.2.2 To scan local files or directories, execute the following command:

```bash
java -jar lpvs-*.jar --local.path=</path/to/file/or/folder>
```

4.2.3 By default, the above commands require a pre-configured MySQL database. Use the "singlescan" profile to skip setting up a pre-configured MySQL database:

```bash
java -jar -Dspring.profiles.active=singlescan -Dgithub.token=<my-token> lpvs-*.jar --github.pull.request=<PR URL>
java -jar -Dspring.profiles.active=singlescan lpvs-*.jar --local.path=</path/to/file/or/folder>
```

These steps streamline the process, allowing you to run a scan on a single pull request without the need for a preconfigured database.
4.2.4 Optionally, generate an HTML report and save it in a specified folder. Replace `path/to/your/folder` with the full path to the folder where you want to save the HTML report, and `your_report_filename.html` with the desired filename for the report.

4.2.4 Available option to generate an HTML report and save it in a specified folder. Replace `/path/to/your/folder` with the full path to the folder where you want to save the HTML report, and `your_report_filename.html` with the desired filename for the report.
```bash
java -jar -Dgithub.token=<my-token> lpvs-*.jar --github.pull.request=<PR URL> --build.html.report=</path/to/your/folder/your_report_filename.html>
java -jar -Dspring.profiles.active=singlescan -Dgithub.token=<my-token> lpvs-*.jar --github.pull.request=<PR URL> --build.html.report=</path/to/your/folder/your_report_filename.html>
java -jar -Dspring.profiles.active=singlescan lpvs-*.jar --local.path=</path/to/file/or/folder> --build.html.report=<your_report_filename.html>
```

These steps streamline the process, allowing you to run a scan on a single pull request without the need for a preconfigured database.
Note: Ensure that the specified folder exists before generating the HTML report.

4.2.5 Examples of commands:

```bash
java -jar -Dspring.profiles.active=singlescan lpvs-*.jar --github.pull.request=https://github.com/Samsung/LPVS/pull/2
java -jar -Dspring.profiles.active=singlescan lpvs-*.jar --github.pull.request=https://github.com/Samsung/LPVS/pull/2 --build.html.report=report.html
java -jar -Dspring.profiles.active=singlescan lpvs-*.jar --local.path=test.c
java -jar -Dspring.profiles.active=singlescan lpvs-*.jar --local.path=test --build.html.report=test/report.html
```

#### 4.3 Use of _LPVS_ JAR `lpvs-x.y.z.jar` in your project

Expand Down
13 changes: 4 additions & 9 deletions src/main/java/com/lpvs/service/LPVSLicenseService.java
Original file line number Diff line number Diff line change
Expand Up @@ -258,15 +258,10 @@ public LPVSLicense getLicenseBySpdxIdAndName(
lic = findLicenseInOsoriDB(licenseSpdxId);
// If not found, create new license with default field values
if (lic == null) {
lic =
new LPVSLicense() {
{
setSpdxId(licenseSpdxId);
setLicenseName(licName);
setAlternativeNames(null);
setAccess("UNREVIEWED");
}
};
lic = new LPVSLicense();
lic.setSpdxId(licenseSpdxId);
lic.setLicenseName(licName);
lic.setAccess("UNREVIEWED");
}
// Save new license
lic = lpvsLicenseRepository.saveAndFlush(lic);
Expand Down
13 changes: 10 additions & 3 deletions src/main/java/com/lpvs/service/LPVSQueueProcessorService.java
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,17 @@ public class LPVSQueueProcessorService {
private LPVSQueueService queueService;

/**
* Trigger value obtained from application properties.
* Trigger value to start a single scan of a pull request (optional).
*/
@Value("${github.pull.request:}")
private String trigger;

/**
* Trigger value to start a single scan of local files or folder (optional).
*/
@Value("${local.path:}")
private String localPath;

/**
* Constructor for LPVSQueueProcessorService.
*
Expand All @@ -51,12 +57,13 @@ public class LPVSQueueProcessorService {
* @throws Exception If an exception occurs during queue processing.
*/
@EventListener(ApplicationReadyEvent.class)
private void queueProcessor() throws Exception {
protected void queueProcessor() throws Exception {
// Check for any pending elements in the LPVSQueue.
queueService.checkForQueue();

// Process LPVSQueue elements until the trigger is set.
while (trigger == null || trigger.isEmpty()) {
while ((trigger == null || trigger.isBlank())
&& (localPath == null || localPath.isBlank())) {
Copy link
Collaborator

@t-naumenko t-naumenko Aug 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

localPath.isBlank() includes a case with null, right?
If so, there is no reason to check it for localPath and trigger:

 while (trigger.isBlank())
                && (localPath.isBlank())
                ...

// Get the first element from the LPVSQueue.
LPVSQueue webhookConfig = queueService.getQueueFirstElement();
log.info("PROCESS Webhook id = " + webhookConfig.getId());
Expand Down
140 changes: 106 additions & 34 deletions src/main/java/com/lpvs/service/scan/LPVSDetectService.java
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,15 @@
package com.lpvs.service.scan;

import java.io.File;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;

import com.lpvs.service.LPVSGitHubConnectionService;
import com.lpvs.service.LPVSGitHubService;
import com.lpvs.service.LPVSLicenseService;
import com.lpvs.util.LPVSFileUtil;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.SpringApplication;
Expand Down Expand Up @@ -59,11 +59,17 @@ public class LPVSDetectService {
private LPVSScanService scanService;

/**
* GitHub pull request used to trigger a single license scan (optional).
* Trigger value to start a single scan of a pull request (optional).
*/
@Value("${github.pull.request:}")
private String trigger;

/**
* Trigger value to start a single scan of local files or folder (optional).
*/
@Value("${local.path:}")
private String localPath;

/**
* Optional parameter to save html report to specified location.
*/
Expand Down Expand Up @@ -105,52 +111,118 @@ public LPVSDetectService(
*/
@EventListener(ApplicationReadyEvent.class)
public void runSingleScan() {
if (trigger != null && !HtmlUtils.htmlEscape(trigger).isEmpty()) {
log.info("Triggered single scan operation");
// generateReport indicates that a report should be generated (HTML or command line output)
boolean generateReport = false;
LPVSQueue webhookConfig = null;
List<LPVSFile> scanResult = null;
List<LPVSLicenseService.Conflict<String, String>> detectedConflicts = null;

// Error case when both pull request scan and local files scan are set to true
if (trigger != null && !trigger.isBlank() && localPath != null && !localPath.isBlank()) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As isBlank include check for a null, so it could be simplified to:
if (!trigger.isBlank() && !localPath.isBlank())

log.error(
"Incorrect settings: both pull request scan and local files scan are set to true.");
SpringApplication.exit(ctx, () -> 0);

// Scan option - single pull request scan
} else if (trigger != null && !HtmlUtils.htmlEscape(trigger).isBlank()) {
log.info("Triggered single scan of pull request.");
try {
licenseService.reloadFromTables();
LPVSQueue webhookConfig =
webhookConfig =
gitHubService.getInternalQueueByPullRequest(HtmlUtils.htmlEscape(trigger));

List<LPVSFile> scanResult =
scanResult =
this.runScan(
webhookConfig, gitHubService.getPullRequestFiles(webhookConfig));

List<LPVSLicenseService.Conflict<String, String>> detectedConflicts =
licenseService.findConflicts(webhookConfig, scanResult);

if (htmlReport != null && !HtmlUtils.htmlEscape(htmlReport).isEmpty()) {
Path buildReportPath = Paths.get(htmlReport);
Path parentDirectory = buildReportPath.getParent();

if (parentDirectory != null && Files.isDirectory(parentDirectory)) {
String report =
LPVSCommentUtil.buildHTMLComment(
webhookConfig, scanResult, detectedConflicts);
LPVSCommentUtil.saveHTMLToFile(report, buildReportPath.toString());
} else {
log.error(
"Error: The parent directory '"
+ parentDirectory
+ "' does not exist.");
}
detectedConflicts = licenseService.findConflicts(webhookConfig, scanResult);
generateReport = true;
log.info("Single scan of pull request completed.");
} catch (Exception ex) {
log.error("Single scan of pull request finished with errors.");
log.error("Can't trigger single scan: " + ex.getMessage());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Single scan finished" and at the same time "Can't trigger single scan"
What about use something general: ""
log.error("Single scan of pull request failed with error: " + ex.getMessage());

SpringApplication.exit(ctx, () -> 0);
}

// Scan option - single scan of local file or folder
} else if (localPath != null && !HtmlUtils.htmlEscape(localPath).isEmpty()) {
log.info("Triggered single scan of local file(s).");
try {
licenseService.reloadFromTables();
File localFile = new File(localPath);
if (localFile.exists()) {
// 1. Generate webhook config
webhookConfig = getInternalQueueByLocalPath();
// 2. Copy files
LPVSFileUtil.copyFiles(
localPath, LPVSFileUtil.getLocalDirectoryPath(webhookConfig));
// 3. Trigger scan
scanResult =
this.runScan(
webhookConfig,
LPVSFileUtil.getLocalDirectoryPath(webhookConfig));

detectedConflicts = licenseService.findConflicts(webhookConfig, scanResult);
generateReport = true;
log.info("Single scan of local file(s) completed.");
} else {
String report =
LPVSCommentUtil.reportCommentBuilder(
webhookConfig, scanResult, detectedConflicts);
if (report != null && !report.isEmpty()) {
log.info(report);
}
throw new Exception("File path does not exist: " + localPath);
}
log.info("Single scan completed.");

} catch (Exception ex) {
log.error("Single scan finished with errors.");
log.error("Single scan of local file(s) finished with errors.");
log.error("Can't trigger single scan: " + ex.getMessage());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about:
log.error("Single scan of local file(s) failed with error: " + ex.getMessage());

SpringApplication.exit(ctx, () -> 0);
}
}

// Report generation
// 1. HTML format
if (generateReport && htmlReport != null && !HtmlUtils.htmlEscape(htmlReport).isEmpty()) {
File report = new File(htmlReport);
String folderPath = report.getParent();
if (folderPath == null) {
folderPath = ".";
}
File folder = new File(folderPath);
if (folder.exists() && folder.isDirectory()) {
String reportFile =
LPVSCommentUtil.buildHTMLComment(
webhookConfig, scanResult, detectedConflicts);
LPVSCommentUtil.saveHTMLToFile(reportFile, report.getAbsolutePath());
} else {
log.error("Error: The parent directory '" + folder.getPath() + "' does not exist.");
}
SpringApplication.exit(ctx, () -> 0);
} else if (generateReport) {
// 2. Command line output
String report =
LPVSCommentUtil.reportCommentBuilder(
webhookConfig, scanResult, detectedConflicts);
if (!report.isEmpty()) {
log.info(report);
}
SpringApplication.exit(ctx, () -> 0);
}
}

/**
* Creates a new LPVSQueue object with default values for a local scan.
*
* @return the new LPVSQueue object
*/
private LPVSQueue getInternalQueueByLocalPath() {
LPVSQueue queue = new LPVSQueue();
queue.setDate(new Date());
queue.setUserId("Single scan of local files run");
queue.setReviewSystemType("local_scan");
SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMdd-HHmmss");
String repoUrl = "local_scan_" + sdf.format(queue.getDate());
queue.setRepositoryUrl(repoUrl);
queue.setPullRequestUrl(repoUrl);
return queue;
}

/**
* Runs a license scan based on the selected scanner type.
*
Expand Down
38 changes: 38 additions & 0 deletions src/main/java/com/lpvs/util/LPVSFileUtil.java
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import java.util.Arrays;
import java.util.List;

Expand Down Expand Up @@ -125,6 +126,43 @@ public static String saveGithubDiffs(
return directoryPath;
}

/**
* Copies files from a source path to a destination directory path.
*
* @param sourcePath the path to the source file or directory to be copied
* @param directoryPath the path to the destination directory where the files will be copied
* @throws IOException if an I/O error occurs
*/
public static void copyFiles(String sourcePath, String directoryPath) throws IOException {
deleteIfExists(directoryPath);
File destination = new File(directoryPath);
File source = new File(sourcePath);
if (destination.mkdirs()) {
if (source.isDirectory()) {
File[] files = source.listFiles();
if (files != null) {
for (File file : files) {
if (file.isFile()) {
Files.copy(
file.toPath(),
new File(destination, file.getName()).toPath(),
StandardCopyOption.REPLACE_EXISTING);
} else if (file.isDirectory()) {
File destinationSubdir = new File(destination, file.getName());
destinationSubdir.mkdirs();
copyFiles(file.getAbsolutePath(), destinationSubdir.getAbsolutePath());
}
}
}
} else if (source.isFile()) {
Files.copy(
source.toPath(),
new File(destination, source.getName()).toPath(),
StandardCopyOption.REPLACE_EXISTING);
}
}
}

/**
* Deletes the specified directory if it exists.
*
Expand Down
Loading
Loading