devonfw#1529 added first implementation of extensible artifact search…

… REST API (devonfw#1556)

* devonfw#1529 added first implementation of extensible artifact search REST API
added new getMavenArtifactsByGroupId to MavenUtils which takes a repository type (e.g. maven, nexus, jfrog) and a groupId to search for
added new SearchResponse tos to core-api utils
added new MavenUtilTest class (tests for proper json string parsing and REST API requests) and resources

* devonfw#1529 added missing dependencies

* devonfw#1529 implemented requested changes
added new RESTSearchResponseException
replaced IOException with RESTSearchResponseException

* devonfw#1529 fixed jFrog API

* devonfw#1529 implemented requested changes
changed repositoryType from String to MavenSearchRepositoryType enum
added new MavenSearchRepositoryType enum

* devonfw#1529 implemented requested changes
replaced fixed repository URLs with constants
added new MavenSearchRepositoryConstants

* devonfw#1529 implemented requested changes
converted MavenSearchResponseConstants to uppercase
added MAVEN, NEXUS and JFROG TARGET_LINK constants
added MAVEN_MAX_RESPONSE_ROWS constant
added javadoc to all models (moved main model to top)
renamed json response models

* devonfw#1529 adjusted nexus REST API
adjusted nexus REST API to v2.0
adjusted nexus REST API tests and resources to v2.0
refactored createDownloadLink method (moved to AbstractRESTSearchResponse)
added new constants for MAVEN_REPOSITORY_LINK, NEXUS_REPOSITORY_URL, NEXUS_REPOSITORY_URL, NEXUS_TARGET_LINK, NEXUS_DC_ID
added ec param to MavenSearchResponse

* devonfw#1529 disabled jfrog REST API test

* devonfw#1529 added optimizations + authentication
added bearer token authentication
added bearer token to jfrog
renamed nexus to nexus2 repository type
added nexus3 repository type
added more tests for code coverage

* devonfw#1529 applied factory pattern
added new SearchResponse interface
made all SearchResponse types inherit from SearchResponse
added new SearchResponseFactory
refactored getArtifactDownloadLinks method
added new getAvailableSearchInterfaces method (used to register new search interfaces)
added getRepositoryType to SearchResponse parent class (returns the type of repository as an enum)

* devonfw#1529 moved utility methods from factory class to util class
added new SearchResponseUtil class

* changed javadoc

* devonfw#1529 implemented requested changes
renamed RESTSearchResponse signature msg to message

* devonfw#1529 implemented requested changes
removed limitRows functionality
removed NEXUS2_DC_ID from target link
adjusted MavenSearchResponseConstants javadoc

* devonfw#1529 implemented requested changes
converted getAvailableSearchInterfaces method to final list of SearchResponses
adjusted javadoc

* devonfw#1529 added missing javadoc

* devonfw#1529 implemented requested changes
added more detailed javadoc

* devonfw#1529 implemented requested changes
converted RESTSearchResponseException name to PascalCase

* devonfw#1529 implemented requested changes
renamed getMavenArtifactsByGroupId to retrieveMavenArtifactsByGroupId

* devonfw#1529 implemented requested changes
renamed getArtifactDownloadLinks to searchArtifactDownloadLinks

* devonfw#1529 implemented requested changes
moved multi search response classes to separate packages

* devonfw#1529 implemented requested changes
improved javadoc readability

* devonfw#1529 implemented requested changes
converted SearchResponse to AbstractSearchResponse
refactored getJsonResponse
added new retrieveJsonResponseWithAuthentication method to AbstractSearchResponse
renamed getJsonResponse to retrieveJsonResponse

* devonfw#1529 implemented requested changes
refactored getDownloadURLs method
added new removeDuplicatedDownloadURLs method to AbstractSearchResponse
renamed getDownloadURLs to retrieveDownloadURLs

* devonfw#1529 implemented requested changes
refactored SearchResponseUtil methods
moved SearchResponseUtil methods to AbstractSearchResponse
removed SearchResponseUtil class

* devonfw#1529 implemented requested changes
removed throw CobiGenRuntimeExceptions to ensure that an error with the API won't stop CobiGen execution
replaced CobiGenRuntimeException in MavenUtil with error log and a return of null
converted testWrongRepositoryTypeThrowsException to testRetrieveMavenArtifactsWithInvalidLinkReturnsNull
replaced throw exception with error log message

* devonfw#1529 implemented requested changes
refactored getJsonResponseStringByTargetLink
replaced getJsonResponseStringByTargetLink with retrieveJsonResponseWithAuthenticationToken
added MavenSearchRepositoryType to retrieveJsonResponseWithAuthenticationToken

* devonfw#1529 implemented requested changes
replaced fixed ignored json properties with ignore unknown param

* devonfw#1529 added WireMock to tests
made ignored tests functional with WireMock
added WireMock stubs for each case
updated jersey from 3.0.5 to 3.0.7
added wiremock-standalone 2.27.2 (used older version because of conflicts with jackson)

* devonfw#1529 reduced WireMock logging
added logback-test.xml (sets WireMock to WARN log level)

* devonfw#1529 updated jackson
updated jackson-databind from 2.13.2.2 to 2.13.3

* devonfw#1529 fixed FileUtils issue
replaced readString with readAllBytes

* devonfw#1529 implemented requested changes
renamed baseURL to baseUrl
removed LOG concatenation

* devonfw#1529 implemented requested changes
removed ProcessingException from SearchResponseFactory

* devonfw#1529 implemented requested changes
added artifactory path to jfrog target link constant

* devonfw#1529 removed jersey dependencies
replaced jersey with OkHttpClient
restricted MavenUtilTests (added check of messages)
added jackson-databind to core-api dependencies
added okhttp to core-api dependencies

* devonfw#1529 updated okhttp
changed okhttp version from 4.9.1 to 4.10.0

* devonfw#1529 added okhttp version to root pom
added latest okhttp dependeny to root pom.xml
removed fixed okhttp versions from core-api and core-externalprocess-api

* devonfw#1529 added okio
added transitive okio dependency

* devonfw#1529 implemented requested changes
Changed enum values to uppercase

* devonfw#1529 implemented requested changes
moved status into not null if condition

Co-authored-by: EduardKrieger <eduard.krieger@capgemini.com>

cobigen/cobigen-core-api/pom.xml

@@ -27,7 +27,7 @@
       <groupId>com.google.guava</groupId>
       <artifactId>guava</artifactId>
     </dependency>
-    
+
     <!-- Proper process reader -->
     <dependency>
       <groupId>org.zeroturnaround</groupId>
@@ -40,11 +40,32 @@
       <artifactId>core-test</artifactId>
       <scope>test</scope>
     </dependency>
+
     <dependency>
       <groupId>commons-io</groupId>
       <artifactId>commons-io</artifactId>
       <scope>test</scope>
     </dependency>
+
+    <!-- enables JSON creation for REST search response API -->
+    <dependency>
+      <groupId>com.fasterxml.jackson.core</groupId>
+      <artifactId>jackson-databind</artifactId>
+    </dependency>
+
+    <!-- http client -->
+    <dependency>
+      <groupId>com.squareup.okhttp3</groupId>
+      <artifactId>okhttp</artifactId>
+    </dependency>
+
+    <!-- Needed for WireMock test support -->
+    <dependency>
+      <groupId>com.github.tomakehurst</groupId>
+      <artifactId>wiremock-standalone</artifactId>
+      <version>2.27.2</version>
+      <scope>test</scope>
+    </dependency>
   </dependencies>
 
   <build>

cobigen/cobigen-core-api/src/main/java/com/devonfw/cobigen/api/constants/MavenSearchRepositoryConstants.java

@@ -0,0 +1,58 @@
+package com.devonfw.cobigen.api.constants;
+
+/**
+ * Constants needed for handling the maven search REST APIs
+ */
+public class MavenSearchRepositoryConstants {
+
+  /**
+   * Maven repository URL
+   */
+  public static String MAVEN_REPOSITORY_URL = "https://search.maven.org";
+
+  /**
+   * Maven repository download link
+   */
+  public static String MAVEN_REPOSITORY_DOWNLOAD_LINK = "https://repo1.maven.org/maven2";
+
+  /**
+   * Maven target link
+   */
+  public static String MAVEN_TARGET_LINK = "solrsearch/select";
+
+  /**
+   * Nexus2 repository URL
+   */
+  public static String NEXUS2_REPOSITORY_URL = "https://s01.oss.sonatype.org";
+
+  /**
+   * Nexus2 repository link
+   */
+  public static String NEXUS2_REPOSITORY_LINK = "service/local/repositories/releases/content";
+
+  /**
+   * Nexus2 target link
+   */
+  public static String NEXUS2_TARGET_LINK = "service/local/lucene/search";
+
+  /**
+   * Nexus3 target link
+   */
+  public static String NEXUS3_TARGET_LINK = "service/rest/v1/search";
+
+  /**
+   * Nexus3 repository URL
+   */
+  public static String NEXUS3_REPOSITORY_URL = "";
+
+  /**
+   * Jfrog repository URL
+   */
+  public static String JFROG_REPOSITORY_URL = "http://localhost:8082/artifactory";
+
+  /**
+   * Jfrog target link
+   */
+  public static String JFROG_TARGET_LINK = "artifactory/api/search/gavc";
+
+}

cobigen/cobigen-core-api/src/main/java/com/devonfw/cobigen/api/constants/MavenSearchRepositoryType.java

@@ -0,0 +1,28 @@
+package com.devonfw.cobigen.api.constants;
+
+/**
+ * Maven search repository types used to identify and name the available search REST API types (add new search
+ * repository types/versions here)
+ */
+public enum MavenSearchRepositoryType {
+
+  /**
+   * Nexus2 search repository type
+   */
+  NEXUS2,
+
+  /**
+   * Nexus3 search repository type
+   */
+  NEXUS3,
+
+  /**
+   * Maven search repository type
+   */
+  MAVEN,
+
+  /**
+   * Jfrog search repository type
+   */
+  JFROG
+}

cobigen/cobigen-core-api/src/main/java/com/devonfw/cobigen/api/exception/RestSearchResponseException.java

@@ -0,0 +1,40 @@
+package com.devonfw.cobigen.api.exception;
+
+/** Exception to indicate that the REST search API encountered a problem while accessing the server. */
+public class RestSearchResponseException extends CobiGenRuntimeException {
+
+  private static final long serialVersionUID = 1L;
+
+  /**
+   * Creates a new {@link RestSearchResponseException} with the given message
+   *
+   * @param message error message of the exception
+   */
+  public RestSearchResponseException(String message) {
+
+    super(message);
+  }
+
+  /**
+   * Creates a new {@link RestSearchResponseException} with the specified message and the causing {@link Throwable}
+   *
+   * @param message describing the exception
+   * @param cause the causing Throwable
+   */
+  public RestSearchResponseException(String message, Throwable cause) {
+
+    super(message, cause);
+  }
+
+  /**
+   * Creates a new {@link RestSearchResponseException} with the specified message and the causing {@link Throwable}
+   *
+   * @param message describing the exception
+   * @param statusCode status code causing the {@link RestSearchResponseException} or null if not available
+   */
+  public RestSearchResponseException(String message, String statusCode) {
+
+    super(message + statusCode);
+  }
+
+}

cobigen/cobigen-core-api/src/main/java/com/devonfw/cobigen/api/util/MavenUtil.java

@@ -27,6 +27,9 @@
 
 import com.devonfw.cobigen.api.constants.MavenConstants;
 import com.devonfw.cobigen.api.exception.CobiGenRuntimeException;
+import com.devonfw.cobigen.api.exception.RestSearchResponseException;
+import com.devonfw.cobigen.api.util.to.SearchResponseFactory;
+import com.fasterxml.jackson.core.JsonProcessingException;
 import com.google.common.collect.Lists;
 import com.google.common.hash.Hashing;
 import com.google.common.io.ByteSource;
@@ -340,4 +343,27 @@ public static Path getProjectRoot(Path inputFile, boolean topLevel) {
     LOG.debug("Project root could not be found.");
     return null;
   }
+
+  /**
+   * Retrieves a list of download URLs by groupId from the specified repository search REST API using authentication
+   * with bearer token
+   *
+   * @param baseUrl String of the repository server URL
+   * @param groupId the groupId to search for
+   * @param authToken bearer token to use for authentication
+   * @return List of artifact download URLS or null if an error occurred
+   */
+  public static List<URL> retrieveMavenArtifactsByGroupId(String baseUrl, String groupId, String authToken) {
+
+    try {
+
+      return SearchResponseFactory.searchArtifactDownloadLinks(baseUrl, groupId, authToken);
+    } catch (RestSearchResponseException | JsonProcessingException | MalformedURLException e) {
+      LOG.error("Unable to get artifacts from {} by groupId {}", baseUrl, groupId, e);
+      // TODO: Handle Eclipse, CLI and MavenPlugin here (f.e. with a new Exception)
+      return null;
+    }
+
+  }
+
 }

cobigen/cobigen-core-api/src/main/java/com/devonfw/cobigen/api/util/to/AbstractSearchResponse.java

@@ -0,0 +1,164 @@
+package com.devonfw.cobigen.api.util.to;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.devonfw.cobigen.api.constants.MavenSearchRepositoryType;
+import com.devonfw.cobigen.api.exception.RestSearchResponseException;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+import okhttp3.OkHttpClient;
+import okhttp3.Request;
+import okhttp3.Response;
+
+/**
+ * This interface should be inherited for all maven REST search API responses to properly convert {@link JsonProperty}
+ * from responses to valid download URLs
+ */
+public abstract class AbstractSearchResponse {
+
+  /** Logger instance. */
+  @JsonIgnore
+  private static final Logger LOG = LoggerFactory.getLogger(AbstractSearchResponse.class);
+
+  /**
+   * @return the {@link MavenSearchRepositoryType} type
+   */
+  public abstract MavenSearchRepositoryType getRepositoryType();
+
+  /**
+   * Creates a list of download URLs
+   *
+   * @return List of download links
+   * @throws MalformedURLException if an URL was not valid
+   */
+  public abstract List<URL> retrieveDownloadURLs() throws MalformedURLException;
+
+  /**
+   * Removes duplicates from list of download URLs
+   *
+   * @param downloadUrls list of download URLs
+   * @return List of download links
+   * @throws MalformedURLException if an URL was not valid
+   */
+  public List<URL> removeDuplicatedDownloadURLs(List<URL> downloadUrls) throws MalformedURLException {
+
+    return downloadUrls.stream().distinct().collect(Collectors.toList());
+  }
+
+  /**
+   * Retrieves the json response from a repository URL and a group ID
+   *
+   * @param repositoryUrl URL of the repository
+   * @param groupId to search for
+   * @return String of json response
+   * @throws RestSearchResponseException if the request did not return status 200
+   */
+  public String retrieveJsonResponse(String repositoryUrl, String groupId) throws RestSearchResponseException {
+
+    return retrieveJsonResponse(repositoryUrl, groupId, null);
+  }
+
+  /**
+   * Retrieves the json response from a repository URL, a group ID and a bearer authentication token
+   *
+   * @param repositoryUrl URL of the repository
+   * @param groupId to search for
+   * @param authToken bearer token to use for authentication
+   * @return String of json response
+   * @throws RestSearchResponseException if the request did not return status 200
+   */
+  public abstract String retrieveJsonResponse(String repositoryUrl, String groupId, String authToken)
+      throws RestSearchResponseException;
+
+  /**
+   * Creates a @WebTarget with provided authentication token
+   *
+   * @param targetLink link to get response from
+   * @param token bearer token to use for authentication
+   * @return Request to use as resource
+   */
+  private static Request bearerAuthenticationWithOAuth2AtClientLevel(String targetLink, String token) {
+
+    return new Request.Builder().url(targetLink).addHeader("Authorization", "Bearer " + token).build();
+
+  }
+
+  /**
+   * Retrieves a json response by given REST API target link using bearer authentication token
+   *
+   * @param targetLink link to get response from
+   * @param authToken bearer token to use for authentication
+   * @param searchRepositoryType the type of the search repository
+   * @return String of json response
+   * @throws RestSearchResponseException if the returned status code was not 200 OK
+   */
+  public static String retrieveJsonResponseWithAuthenticationToken(String targetLink, String authToken,
+      MavenSearchRepositoryType searchRepositoryType) throws RestSearchResponseException {
+
+    LOG.info("Starting {} search REST API request with URL: {}.", searchRepositoryType, targetLink);
+
+    OkHttpClient httpClient = new OkHttpClient().newBuilder().connectTimeout(10, TimeUnit.SECONDS)
+        .readTimeout(30, TimeUnit.SECONDS).callTimeout(30, TimeUnit.SECONDS).writeTimeout(30, TimeUnit.SECONDS)
+        .retryOnConnectionFailure(true).build();
+    String jsonResponse = "";
+
+    try {
+      Response response = null;
+
+      if (authToken != null) {
+        response = httpClient.newCall(bearerAuthenticationWithOAuth2AtClientLevel(targetLink, authToken)).execute();
+      } else {
+        response = httpClient.newCall(new Request.Builder().url(targetLink).get().build()).execute();
+      }
+
+      if (response != null) {
+        int status = response.code();
+        if (status == 200 || status == 201 || status == 204) {
+          jsonResponse = response.body().string();
+        } else {
+          throw new RestSearchResponseException("The search REST API returned the unexpected status code: ",
+              String.valueOf(response.code()));
+        }
+      }
+
+    } catch (IOException e) {
+      throw new RestSearchResponseException("Unable to send or receive the message from the service", e);
+    } catch (IllegalArgumentException e) {
+      throw new RestSearchResponseException("The target URL was faulty.", e);
+    }
+
+    return jsonResponse;
+
+  }
+
+  /**
+   * Creates a download link (concatenates maven repository link with groupId, artifact and version)
+   *
+   * @param mavenRepo link to the maven repository to use
+   * @param groupId for the download link
+   * @param artifactId for the download link
+   * @param version for the download link
+   * @param fileEnding file ending for the download link
+   * @return concatenated download link
+   * @throws MalformedURLException if the URL was not valid
+   */
+  protected static URL createDownloadLink(String mavenRepo, String groupId, String artifactId, String version,
+      String fileEnding) throws MalformedURLException {
+
+    String parsedGroupId = groupId.replace(".", "/");
+    String downloadFile = artifactId + "-" + version + fileEnding;
+    String downloadLink = mavenRepo + "/" + parsedGroupId + "/" + artifactId + "/" + version + "/" + downloadFile;
+    URL url = new URL(downloadLink);
+    return url;
+  }
+
+}