Adding support for DynamoDB backend #85

citizenken · 2018-04-20T03:44:46Z

What

This PR implements a DynamoDB backend for FailureCause storage. It makes the following changes:

Extends KnowledgeBase for Dynamo, inspired by the current Mongo implementation
Adds annotations/a few extra functions to FailureCause, FailureCauseModification, and Indication to facilitate mapping models to Dynamo models/objects
UI-base configuration options
Extensive unit tests

From a user's perspective, the UI/modifying failures/how failures are displayed is unchanged.
From a DB perspective, I have ensured that the data structure is the same as the Mogo structure:

{  
   "_id":"5ad00f8160b2ee54a8330754",
   "name":"foo",
   "description":"bar",
   "comment":"",
   "indications":[  
      {  
         "@class":"com.sonyericsson.jenkins.plugins.bfa.model.indication.BuildLogIndication",
         "pattern":"baz"
      }
   ],
   "modifications":[  
      {  
         "user":"kpetti",
         "time":"2018-04-13T02:01:37.284Z"
      }
   ],
   "lastOccurred":"1970-01-01T00:00:00Z"
}

Why

When evaluating the plugin for use, my team really liked the concept of a dedicated database for tracking failure info. However, without more support outside of our team, we could not justify the operationalizing/maintenance cost of a MongoDB cluster. We're not DBA's, so it falls out of our wheelhouse. Additionally, the plugin uses the mongo-java-driver 2.x, which only communicates with MongoDB v2.x, which has now been EoL'd.

For a no/low-maintenance NoSQL store, we want to use DynamoDB. It is easy to operationalize, and we don't have to dedicate much time/thought into management of the database. As our company is moving more to AWS services, it seems like a natural fit, and could be a good fit for other teams as well.

Testing

I've written a testsuite for the new backend, as well as tests for any functionality I've added. The backend testsuite has a fairly high level of mocking, mostly because of how much DynamoDB's Java library handles behind the scenes.

ToDo

This is a large PR because it adds the bulk of the functionality (CRUD operations). That said, there is still more to implement to reach Mongo parity:

Caching layer for larger queries
Statistic tracking
Result sorting
Investigate DynamoDB scan vs query and which is more appropriate

UI screenshot

rsandell · 2018-06-05T09:38:55Z

pom.xml

@@ -2,7 +2,7 @@
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.sonyericsson.jenkins.plugins.bfa</groupId>
    <artifactId>build-failure-analyzer</artifactId>
-    <version>1.19.3-SNAPSHOT</version>
+    <version>1.20.0</version>


Please don't change this. I'll update the version when releasing the plugin

rsandell · 2018-06-05T09:40:26Z

src/main/java/com/sonyericsson/jenkins/plugins/bfa/db/DynamoDBKnowledgeBase.java

+     */
+    @DataBoundConstructor
+    public DynamoDBKnowledgeBase(String region, String credentialsPath, String credentialsProfile) {
+        if (region == null || region.isEmpty()) {


nit StringUtils.isEmpty() could be used instead.

rsandell · 2018-06-05T09:48:28Z

src/main/java/com/sonyericsson/jenkins/plugins/bfa/db/DynamoDBKnowledgeBase.java

+            convertFromAbstract(oldKnowledgeBase);
+            convertRemoved((DynamoDBKnowledgeBase)oldKnowledgeBase);
+        } else {
+            for (FailureCause cause : oldKnowledgeBase.getCauseNames()) {


getCauseNames() doesn't give you all the data, so the conversion will loose information.

rsandell · 2018-06-05T09:53:24Z

src/main/java/com/sonyericsson/jenkins/plugins/bfa/db/DynamoDBKnowledgeBase.java

+    @Override
+    public int hashCode() {
+        //Making checkstyle happy.
+        return getClass().getName().hashCode();


This doesn't conform to the "equals and hashCode" style, i.e. if two objects are equal they should also have the same hashcode and if two Object's hashcodes are equal the Objects should also be equal.
So add region, credentialsPath and credentialsProfile as well. Most moders IDEs should be able to do that for you.

rsandell · 2018-06-05T09:56:24Z

src/main/java/com/sonyericsson/jenkins/plugins/bfa/db/DynamoDBKnowledgeBase.java

+         * @param credentialProfile the credential profile to use.
+         * @return {@link FormValidation#ok() } if can be done,
+         *         {@link FormValidation#error(java.lang.String) } otherwise.
+         */


@RequirePOST see this week's security advisory many plugins have this issue ;)

rsandell · 2018-06-05T10:05:32Z

src/main/java/com/sonyericsson/jenkins/plugins/bfa/model/FailureCause.java

+    @Override
+    public int hashCode() {
+        //Making checkstyle happy.
+        return getClass().getName().hashCode();


Same equals and hashCode problems here, but in this case just looking at the id might be enough.

rsandell · 2018-06-05T10:09:57Z

src/main/java/com/sonyericsson/jenkins/plugins/bfa/model/FailureCause.java

+                    Indication indication = Indication.class.cast(obj);
+                    indications.add(indication);
+                } catch (Exception e) {
+                    e.printStackTrace();


Should be properly logged via for example logger.

rsandell · 2018-06-05T10:10:31Z

src/main/java/com/sonyericsson/jenkins/plugins/bfa/model/indication/Indication.java

+    @Override
+    public int hashCode() {
+        //Making checkstyle happy.
+        return getClass().getName().hashCode();


rsandell · 2018-06-05T10:11:08Z

src/main/resources/com/sonyericsson/jenkins/plugins/bfa/db/DynamoDBKnowledgeBase/config.jelly

+<!--
+  ~ The MIT License
+  ~
+  ~ Copyright 2012 Sony Mobile Communications AB. All rights reserved.


should be changed to yours

rsandell · 2018-06-05T10:18:11Z

src/main/java/com/sonyericsson/jenkins/plugins/bfa/db/DynamoDBKnowledgeBase.java

+         * @param value the pattern to check.
+         * @return {@link hudson.util.FormValidation#ok()} if everything is well.
+         */
+        public FormValidation doCheckCredentialsPath(@QueryParameter("value") final String value) {


first off this should use proper credentials via the credentials store instead ( the old mongodb code was done before we had any real knowledge of how Jenkins credentials worked :) )

Second this is a potential security risk as it can be used by anyone with overall/read permission to query the existence of a file on the Jenkins master file system. So needs @RequirePOST and a Jenkins.checkPermission(Jenkins.ADMINISTER). The jelly also needs to be updated so the check is done via POST instead of a GET.

rsandell · 2018-06-05T10:20:43Z

Thanks for the contribution, sorry that I'm taking so much time doing reviews.

rajeev-ranjan7 · 2021-06-15T13:18:20Z

@citizenken Any update on this ? We were also looking towards DynamoDB support for this plugin

rajeev-ranjan7 · 2021-08-31T08:28:13Z

@citizenken Any update on this ? We were also looking towards DynamoDB support for this plugin

Petti, Ken added 16 commits April 15, 2018 06:37

Adding pom updates w/Amazon SDK

0a4bbf5

adding dynamodbknowledgebase + tests

779c8a5

dynamodb-related updates for failurecause

b4844a2

adding more functions for DynamoDB

bbd12d5

adding get categories, refactoring dbmapper

0b87b7e

adding equals method

227a559

adding equals method

1820161

adding tests

b46b97c

adding converting functions

3a9ed5b

more tests

fcf8ff5

fixing style checks

7fac5e9

fixing style and tests

7462216

fixing checkstyle

0af8755

comment change

f763e83

fixing exception throwing

cd5e5a6

fixing style issues

c0284c9

citizenken force-pushed the master branch 2 times, most recently from d58bfff to c0284c9 Compare May 21, 2018 14:50

rsandell requested changes Jun 5, 2018

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding support for DynamoDB backend #85

Adding support for DynamoDB backend #85

citizenken commented Apr 20, 2018

rsandell Jun 5, 2018

rsandell Jun 5, 2018

rsandell Jun 5, 2018

rsandell Jun 5, 2018

rsandell Jun 5, 2018

rsandell Jun 5, 2018

rsandell Jun 5, 2018

rsandell Jun 5, 2018

rsandell Jun 5, 2018

rsandell Jun 5, 2018

rsandell commented Jun 5, 2018

rajeev-ranjan7 commented Jun 15, 2021

rajeev-ranjan7 commented Aug 31, 2021

Adding support for DynamoDB backend #85

Are you sure you want to change the base?

Adding support for DynamoDB backend #85

Conversation

citizenken commented Apr 20, 2018

What

Why

Testing

ToDo

UI screenshot

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rsandell commented Jun 5, 2018

rajeev-ranjan7 commented Jun 15, 2021

rajeev-ranjan7 commented Aug 31, 2021