Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add scanning of local files and folders (single scan) #558

Merged
merged 9 commits into from
Aug 7, 2024
35 changes: 27 additions & 8 deletions doc/quick-start-guide-and-build.md
Original file line number Diff line number Diff line change
Expand Up @@ -245,34 +245,53 @@ _LPVS_ is now built and running. You can create a new pull request or update an

#### 4.2 Single scan mode

Alternatively, you can perform a one-time scan on a specific pull request using the single scan mode. Follow these steps:
Alternatively, you can perform a one-time scan on a specific pull request or local files using the single scan mode. Follow these steps:

4.2.1 Begin by running the installation and navigating to the target directory, similar to the process in service mode (refer to steps 4.1.1 and 4.1.2):
4.2.1 Install and navigate to the target directory as described in step 4.1.1 and 4.1.2:

```bash
mvn clean install
cd target/
```

4.2.2 Execute the single scan with the following command:
4.2.2 Choose either to scan a specific pull request from GitHub or local files.

4.2.2.1 To scan a specific pull request from GitHub, execute the following command:

```bash
java -jar -Dgithub.token=<my-token> lpvs-*.jar --github.pull.request=<PR URL>
```

4.2.3 By default, the above command requires a pre-configured MySQL database. To avoid setting up the database, use the "singlescan" profile:
4.2.2.2 To scan local files or directories, execute the following command:

```bash
java -jar lpvs-*.jar --local.path=</path/to/file/or/folder>
```

4.2.3 By default, the above commands require a pre-configured MySQL database. Use the "singlescan" profile to skip setting up a pre-configured MySQL database:

```bash
java -jar -Dspring.profiles.active=singlescan -Dgithub.token=<my-token> lpvs-*.jar --github.pull.request=<PR URL>
java -jar -Dspring.profiles.active=singlescan lpvs-*.jar --local.path=</path/to/file/or/folder>
```

These steps streamline the process, allowing you to run a scan on a single pull request without the need for a preconfigured database.
4.2.4 Optionally, generate an HTML report and save it in a specified folder. Replace `path/to/your/folder` with the full path to the folder where you want to save the HTML report, and `your_report_filename.html` with the desired filename for the report.

4.2.4 Available option to generate an HTML report and save it in a specified folder. Replace `/path/to/your/folder` with the full path to the folder where you want to save the HTML report, and `your_report_filename.html` with the desired filename for the report.
```bash
java -jar -Dgithub.token=<my-token> lpvs-*.jar --github.pull.request=<PR URL> --build.html.report=</path/to/your/folder/your_report_filename.html>
java -jar -Dspring.profiles.active=singlescan -Dgithub.token=<my-token> lpvs-*.jar --github.pull.request=<PR URL> --build.html.report=</path/to/your/folder/your_report_filename.html>
java -jar -Dspring.profiles.active=singlescan lpvs-*.jar --local.path=</path/to/file/or/folder> --build.html.report=<your_report_filename.html>
```

These steps streamline the process, allowing you to run a scan on a single pull request without the need for a preconfigured database.
Note: Ensure that the specified folder exists before generating the HTML report.

4.2.5 Examples of commands:

```bash
java -jar -Dspring.profiles.active=singlescan lpvs-*.jar --github.pull.request=https://github.com/Samsung/LPVS/pull/2
java -jar -Dspring.profiles.active=singlescan lpvs-*.jar --github.pull.request=https://github.com/Samsung/LPVS/pull/2 --build.html.report=report.html
java -jar -Dspring.profiles.active=singlescan lpvs-*.jar --local.path=test.c
java -jar -Dspring.profiles.active=singlescan lpvs-*.jar --local.path=test --build.html.report=test/report.html
```

#### 4.3 Use of _LPVS_ JAR `lpvs-x.y.z.jar` in your project

Expand Down
16 changes: 6 additions & 10 deletions src/main/java/com/lpvs/service/LPVSLicenseService.java
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
import com.lpvs.util.LPVSExitHandler;

import com.lpvs.util.LPVSPayloadUtil;
import io.micrometer.common.util.StringUtils;
import lombok.NoArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
Expand Down Expand Up @@ -230,7 +231,7 @@ protected LPVSLicense findLicenseByName(String name) {
if (license.getLicenseName().equalsIgnoreCase(name)) {
return license;
}
if (license.getAlternativeNames() != null && !license.getAlternativeNames().isBlank()) {
if (!StringUtils.isBlank(license.getAlternativeNames())) {
String[] names = license.getAlternativeNames().split(",");
for (String n : names) {
if (n.trim().equalsIgnoreCase(name)) {
Expand Down Expand Up @@ -258,15 +259,10 @@ public LPVSLicense getLicenseBySpdxIdAndName(
lic = findLicenseInOsoriDB(licenseSpdxId);
// If not found, create new license with default field values
if (lic == null) {
lic =
new LPVSLicense() {
{
setSpdxId(licenseSpdxId);
setLicenseName(licName);
setAlternativeNames(null);
setAccess("UNREVIEWED");
}
};
lic = new LPVSLicense();
lic.setSpdxId(licenseSpdxId);
lic.setLicenseName(licName);
lic.setAccess("UNREVIEWED");
}
// Save new license
lic = lpvsLicenseRepository.saveAndFlush(lic);
Expand Down
13 changes: 10 additions & 3 deletions src/main/java/com/lpvs/service/LPVSQueueProcessorService.java
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
package com.lpvs.service;

import com.lpvs.entity.LPVSQueue;
import io.micrometer.common.util.StringUtils;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
Expand All @@ -29,11 +30,17 @@ public class LPVSQueueProcessorService {
private LPVSQueueService queueService;

/**
* Trigger value obtained from application properties.
* Trigger value to start a single scan of a pull request (optional).
*/
@Value("${github.pull.request:}")
private String trigger;

/**
* Trigger value to start a single scan of local files or folder (optional).
*/
@Value("${local.path:}")
private String localPath;

/**
* Constructor for LPVSQueueProcessorService.
*
Expand All @@ -51,12 +58,12 @@ public class LPVSQueueProcessorService {
* @throws Exception If an exception occurs during queue processing.
*/
@EventListener(ApplicationReadyEvent.class)
private void queueProcessor() throws Exception {
protected void queueProcessor() throws Exception {
// Check for any pending elements in the LPVSQueue.
queueService.checkForQueue();

// Process LPVSQueue elements until the trigger is set.
while (trigger == null || trigger.isEmpty()) {
while (StringUtils.isBlank(trigger) && StringUtils.isBlank(localPath)) {
// Get the first element from the LPVSQueue.
LPVSQueue webhookConfig = queueService.getQueueFirstElement();
log.info("PROCESS Webhook id = " + webhookConfig.getId());
Expand Down
142 changes: 107 additions & 35 deletions src/main/java/com/lpvs/service/scan/LPVSDetectService.java
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,16 @@
package com.lpvs.service.scan;

import java.io.File;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;

import com.lpvs.service.LPVSGitHubConnectionService;
import com.lpvs.service.LPVSGitHubService;
import com.lpvs.service.LPVSLicenseService;
import com.lpvs.util.LPVSFileUtil;
import io.micrometer.common.util.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.SpringApplication;
Expand Down Expand Up @@ -59,11 +60,17 @@ public class LPVSDetectService {
private LPVSScanService scanService;

/**
* GitHub pull request used to trigger a single license scan (optional).
* Trigger value to start a single scan of a pull request (optional).
*/
@Value("${github.pull.request:}")
private String trigger;

/**
* Trigger value to start a single scan of local files or folder (optional).
*/
@Value("${local.path:}")
private String localPath;

/**
* Optional parameter to save html report to specified location.
*/
Expand Down Expand Up @@ -105,52 +112,117 @@ public LPVSDetectService(
*/
@EventListener(ApplicationReadyEvent.class)
public void runSingleScan() {
if (trigger != null && !HtmlUtils.htmlEscape(trigger).isEmpty()) {
log.info("Triggered single scan operation");
// generateReport indicates that a report should be generated (HTML or command line output)
boolean generateReport = false;
LPVSQueue webhookConfig = null;
List<LPVSFile> scanResult = null;
List<LPVSLicenseService.Conflict<String, String>> detectedConflicts = null;

// Error case when both pull request scan and local files scan are set to true
if (!StringUtils.isBlank(trigger) && !StringUtils.isBlank(localPath)) {
log.error(
"Incorrect settings: both pull request scan and local files scan are set to true.");
SpringApplication.exit(ctx, () -> 0);

// Scan option - single pull request scan
} else if (!StringUtils.isBlank(trigger)) {
log.info("Triggered single scan of pull request.");
try {
licenseService.reloadFromTables();
LPVSQueue webhookConfig =
webhookConfig =
gitHubService.getInternalQueueByPullRequest(HtmlUtils.htmlEscape(trigger));

List<LPVSFile> scanResult =
scanResult =
this.runScan(
webhookConfig, gitHubService.getPullRequestFiles(webhookConfig));

List<LPVSLicenseService.Conflict<String, String>> detectedConflicts =
licenseService.findConflicts(webhookConfig, scanResult);

if (htmlReport != null && !HtmlUtils.htmlEscape(htmlReport).isEmpty()) {
Path buildReportPath = Paths.get(htmlReport);
Path parentDirectory = buildReportPath.getParent();

if (parentDirectory != null && Files.isDirectory(parentDirectory)) {
String report =
LPVSCommentUtil.buildHTMLComment(
webhookConfig, scanResult, detectedConflicts);
LPVSCommentUtil.saveHTMLToFile(report, buildReportPath.toString());
} else {
log.error(
"Error: The parent directory '"
+ parentDirectory
+ "' does not exist.");
}
detectedConflicts = licenseService.findConflicts(webhookConfig, scanResult);
generateReport = true;
log.info("Single scan of pull request completed.");
} catch (Exception ex) {
log.error("Single scan of pull request failed with error: " + ex.getMessage());
SpringApplication.exit(ctx, () -> 0);
}

// Scan option - single scan of local file or folder
} else if (!StringUtils.isBlank(localPath)) {
log.info("Triggered single scan of local file(s).");
try {
licenseService.reloadFromTables();
localPath = HtmlUtils.htmlEscape(localPath);
File localFile = new File(localPath);
if (localFile.exists()) {
// 1. Generate webhook config
webhookConfig = getInternalQueueByLocalPath();
// 2. Copy files
LPVSFileUtil.copyFiles(
localPath, LPVSFileUtil.getLocalDirectoryPath(webhookConfig));
// 3. Trigger scan
scanResult =
this.runScan(
webhookConfig,
LPVSFileUtil.getLocalDirectoryPath(webhookConfig));

detectedConflicts = licenseService.findConflicts(webhookConfig, scanResult);
generateReport = true;
log.info("Single scan of local file(s) completed.");
} else {
String report =
LPVSCommentUtil.reportCommentBuilder(
webhookConfig, scanResult, detectedConflicts);
if (report != null && !report.isEmpty()) {
log.info(report);
}
throw new Exception("File path does not exist: " + localPath);
}
log.info("Single scan completed.");

} catch (Exception ex) {
log.error("Single scan finished with errors.");
log.error("Can't trigger single scan: " + ex.getMessage());
log.error("Single scan of local file(s) failed with error: " + ex.getMessage());
SpringApplication.exit(ctx, () -> 0);
}
}

// Report generation
// 1. HTML format
if (generateReport && !StringUtils.isBlank(htmlReport)) {
File report = new File(HtmlUtils.htmlEscape(htmlReport));
String folderPath = report.getParent();
if (folderPath == null) {
folderPath = ".";
}
File folder = new File(folderPath);
if (folder.exists() && folder.isDirectory()) {
String reportFile =
LPVSCommentUtil.buildHTMLComment(
webhookConfig, scanResult, detectedConflicts);
LPVSCommentUtil.saveHTMLToFile(reportFile, report.getAbsolutePath());
} else {
log.error("Error: The parent directory '" + folder.getPath() + "' does not exist.");
}
SpringApplication.exit(ctx, () -> 0);
} else if (generateReport) {
// 2. Command line output
String report =
LPVSCommentUtil.reportCommentBuilder(
webhookConfig, scanResult, detectedConflicts);
if (!report.isEmpty()) {
log.info(report);
}
SpringApplication.exit(ctx, () -> 0);
}
}

/**
* Creates a new LPVSQueue object with default values for a local scan.
*
* @return the new LPVSQueue object
*/
private LPVSQueue getInternalQueueByLocalPath() {
LPVSQueue queue = new LPVSQueue();
queue.setDate(new Date());
queue.setUserId("Single scan of local files run");
queue.setReviewSystemType("local_scan");
SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMdd-HHmmss");
String repoUrl = "local_scan_" + sdf.format(queue.getDate());
queue.setRepositoryUrl(repoUrl);
queue.setPullRequestUrl(repoUrl);
return queue;
}

/**
* Runs a license scan based on the selected scanner type.
*
Expand Down
Loading
Loading