Apache Ranger plugin #13297

dprophet · 2022-07-21T21:57:49Z

Description

Trino native plugin for Apache Ranger. Its a refactoring and rewrite of the Presto It allows full access controls to be applied to catalogs, schemas, tables, columns, and rows.

Strong data access control technologies are one of the required Trino features to make it Enterprise ready.

Is this change a fix, improvement, new feature, refactoring, or other?
New "Trino" feature. There was originally a PrestoSQL based plugin in Apache Ranger but it got orphaned long ago. This brings back these critical enterprise features into open source Trino.

Trino is a rapidly moving open source project. Apache Ranger moves very slow. Its impossible to maintain a Trino plugin outside of Trino itself. The Trino SystemAccessControl spi interfaces change constantly thus breaking any externally maintained plugin.

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)
Its a plugin that implements io.trino.spi.security.SystemAccessControl. It allows you to write an SQL access control policy in the Apache Ranger UI yet manage the policies by ActiveDirectory/LDAP systems found in typical corporations.

How would you describe this change to a non-technical end user or system administrator?

If you run Trino, as a Data Mesh, ie datalakes connected to datalakes, the first problem you will run into is not all users should be entitled to the entire content of the Data Mesh. This new feature allows you to secure the contents across a vast ocean Data Mesh. If you look at the Data Mesh as a tree, Ranger allows you to protect anything from the roots to the leaves and everything in between.

Related issues, pull requests, and links

Apache Ranger

Documentation

(X) Sufficient documentation is included in this PR.

You will find documentation under plugin/trino-ranger. This will be moved to the standard documentation section later.

Release notes

(X) No release notes entries required.
( ) Release notes entries required with the following suggested text:

Rebase from Trino master

cla-bot · 2022-07-21T21:57:50Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Erik Anderson.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email email@example.com
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

kokosing

Skimmed the code. Initial set of comments.

I think we should have product tests that would prove that KRB authentication is working.

kokosing · 2022-08-02T12:03:59Z

docs/src/main/sphinx/connector/ranger.rst

@@ -0,0 +1,212 @@
+=======================


It is not a connector but system access control

I agree, this should likely live under /docs/src/main/sphinx/security in a file called ranger-access-control.rst.

kokosing · 2022-08-02T12:04:13Z

docs/src/main/sphinx/connector/ranger.rst

+
+To connect to Apache Ranger you need:
+
+* Apache Ranger installed, up and running. Compatible with Ranger v2.1.x, 2.2.x, 2.3.x


How do we verify that?

@dprophet we should add some tests for this. @kokosing recommends something like TestSqlStandardAccessControlChecks and we need some environemnt with additional services, so we could follow EnvMultinodeKafka.

kokosing · 2022-08-02T12:05:17Z

plugin/trino-ranger/conf/ranger-policymgr-ssl.xml

@@ -0,0 +1,49 @@
+<?xml version="1.0"?>


How these files are used?

@dprophet ^^ I assume these files are default confs? If so, we should likely make a tutorial on these. I can also help with this. cc: @mosabua

These files are pure example files. I didnt know what/where to put them.

Yeah, perhaps we could add these for use with a test and later consider making a tutorial with them.

kokosing · 2022-08-02T12:06:30Z

plugin/trino-ranger/pom.xml

+            <artifactId>ranger-plugins-common</artifactId>
+            <version>${dep.ranger.version}</version>
+            <exclusions>
+                <exclusion>


Are you sure that things are still working after exclusions? It looks risky. I think we we should shade libraries instead otherwise we may get runtime errors that class is not present.

@dprophet ^^ This is also something that we will validate with product tests.

Yes. This works after exclusions. If you do a mvn dependency:tree you will see a deep tree where same dependencies (with different) versions exists.

bitsondatadev is correct. This can be ironed out with automated product tests.

kokosing · 2022-08-02T12:07:04Z

plugin/trino-ranger/pom.xml

+                <artifactId>maven-enforcer-plugin</artifactId>
+                <configuration>
+                    <rules>
+                        <requireUpperBoundDeps>


Can you please remove these excludes and fix them instead?

@dprophet ^^

kokosing · 2022-08-02T12:09:08Z

plugin/trino-ranger/src/main/java/io/trino/plugin/ranger/RangerSystemAccessControl.java

+        }
+    }
+
+    private void activatePluginClassLoader()


What is this for? Should you use Plugin thread safe entities, like io.trino.plugin.base.classloader.ClassLoaderSafeConnectorAccessControl?

This is for Ranger plugins, not Trino plugins. All ranger plugins use this same setup. I kept it there because I will allow switching access control systems based on catalogs. In some cases I need a simple REGEX matching based on LDAP/AD not configured in Ranger.

I'm not an expert on Ranger, but I've worked on connecting another service to Ranger in this style (having the implementation be part of the service code rather than part of Ranger). So this is my understanding of Ranger class loading and other intricacies:

Ranger has its own class loader which is related to the way it handles injecting code into Trino (or other services). To clarify, am I talking about the cases when a plugin is built in Ranger and then added to the classpath of another service - the way it's done in this PR is different.

In short, Ranger provides a shim class with no logic in it that implements TrinoAuthorizer and calls out to another implementation class. The implementation class should be only loaded by RangerClassLoader so it can bring in its own dependencies. There is a disconnect between service-specific work (in this case, reading configuration for example) and authorization work.

All that activatePluginClassLoader does is:

public void activate() { preActivateClassLoader.set(Thread.currentThread().getContextClassLoader()); Thread.currentThread().setContextClassLoader(this); }

In this case, it ~~might~~ work to remove RangerClassLoader completely and unite the current shim/implementation architecture since Trino has its own system for handling plugin dependencies.

kokosing · 2022-08-02T12:10:51Z

plugin/trino-ranger/src/main/java/io/trino/plugin/ranger/RangerSystemAccessControlImpl.java

+    @Override
+    public Set<String> filterCatalogs(SystemSecurityContext context, Set<String> catalogs)
+    {
+        LOG.debug("==> RangerSystemAccessControl.filterCatalogs(" + catalogs + ")");


use io.trino.plugin.base.util.LoggingInvocationHandler instead of this logging.

Do you have an example of the LoggingInvocationHandler? There is no usage nor examples in Trino code base. The standard in all the code is io.airlift.log.Logger.

kokosing · 2022-08-02T12:11:58Z

plugin/trino-ranger/src/main/java/io/trino/plugin/ranger/RangerSystemAccessControlImpl.java

+            }
+
+            viewExpression = new ViewExpression(
+                    context.getIdentity().getUser(),


why current user? That means if you have recursive masks or filters then you may end up that you see filtered or masked data in subsequent expression.

In the FileBasedSystemAccessControl.java getRowFilters

They also context.getIdentity().getUser() in the same way I do recall.

DISCLAIMER (off topic): Nesting/recursive row level filters is not a good idea for performance reasons

kokosing · 2022-08-02T12:12:17Z

plugin/trino-ranger/src/main/java/io/trino/plugin/ranger/RangerSystemAccessControlImpl.java

+    }
+
+    @Override
+    public void checkCanKillQueryOwnedBy(SystemSecurityContext context, String queryOwner)


How do you tests this?

I have significant changes not pushed for this PR around this permissioning. I am happy to push but @bitsondatadev recommended I push on the next round. That said, there is a lot I havent pushed but are very necessary to this functionality

In the non-pushed changes:

In Ranger there is a checkbox, impersonate. If that checked, TrinoAccessType.IMPERSONATE is set, you can login to the Trino UI, click on a running query, and kill another users query. Also if you are a superuser you can impersonate any user and kill the running queries.

kokosing · 2022-08-02T12:13:25Z

plugin/trino-ranger/src/test/java/io/trino/plugin/ranger/RangerSystemAccessControlTest.java

+import static io.trino.spi.security.PrincipalType.USER;
+import static io.trino.spi.security.Privilege.SELECT;
+
+public class RangerSystemAccessControlTest


rename to TestRangerSystemAccessControl

It looks like most of the access control methods are not covered.

It looks like most of the access control methods are not covered.
I know.

bitsondatadev

Hey @dprophet, I made a few suggestions around the documentation and highlighted a few comments that @kokosing made that need to be addressed before we loop him back in.

I am happy to work on the devex side of this PR while you address the build and testing issues.

bitsondatadev · 2022-12-14T11:02:43Z

docs/src/main/sphinx/connector/ranger.rst

+
+https://ranger.apache.org/
+
+Apache Ranger is a framework to enable comprehensive data security across many platforms. Trino is one of these platforms.


Suggested change

Apache Ranger is a framework to enable comprehensive data security across many platforms. Trino is one of these platforms.

Apache Ranger is a framework to enable comprehensive data security across many platforms, including Trino.

bitsondatadev · 2022-12-14T11:08:46Z

plugin/trino-ranger/README.md

+This plugin is designed to be build and run inside Trino.
+
+It works with vanilla Apache Ranger, version 2.2.1 and up. You dont need to customize Ranger at all.
+
+Here are the setup steps.
+


Suggested change

This plugin is designed to be build and run inside Trino.

It works with vanilla Apache Ranger, version 2.2.1 and up. You dont need to customize Ranger at all.

Here are the setup steps.

Apache Ranger is a framework to enable comprehensive data security across many platforms, including Trino. Rather than maintain the plugin with the Apache Ranger project, this plugin is designed to be built and run inside Trino.

It works with vanilla Apache Ranger, version 2.2.1 and up. You don't need to customize Ranger at all.

Apache Ranger is trademarked by the ASF, so saying vanilla Apache Ranger seems a bit redundant?

@cbaenziger vanilla Ranger means ZERO changes in Ranger. Works out of the box. Happy to change the wording.

bitsondatadev · 2022-12-14T11:11:37Z

plugin/trino-ranger/README.md

+
+### Full Enterprise ready Apache Ranger setup
+
+ * Install and startup Ranger


I would like to break this up a bit and avoid the "quick and dirty" terminology. Quick may indicate that it should be simple and make someone feel dumb if they get stuck. Dirty may indicate that it's not ready for production.

Happy to help you break this up. We'll also likely want to make some video enablement around this. That will give me a chance to actually run through and validate these steps and any potential hiccups users might encounter.

We'll also likely want to make some video enablement around this.

Ya, I was thinking the same thing. IMO, ASF Ranger community has some poor job of documentation. Worse I have seen.

bitsondatadev · 2022-12-14T11:14:39Z

plugin/trino-ranger/README.md

+           * Most of the defaults work fine. The configs you want to pay attention to are
+             * PYTHON_COMMAND_INVOKER, DB_FLAVOR, SQL_CONNECTOR_JAR, db_root_user, db_root_password, db_host, db_name, db_user, db_password, rangerAdmin_password, rangerTagsync_password, rangerUsersync_password, keyadmin_password
+       * $RANGER_HOME/usersync/install.properties
+         * In most enterprise environments, ActiveDirectory/LDAP are used to manage users and their roles.


We should likely have a separate section for LDAP that we reference here. That would make this section shorter and also leave it up to the reader if they want to just play around they can do as they please. I think the recommendation is sound but not everyone is ready to immediately jump into an LDAP setup.

We should likely have a separate section for LDAP

Agreed. This should be 2 separate setups. LDAP or stand-alone.

bitsondatadev · 2022-12-14T11:16:03Z

plugin/trino-ranger/README.md

+         * User: admin, Password: (defined above in rangerAdmin_password)
+     * Create your Trino service.
+       * The name is important and needs to be configured in the access-control-ranger.properties file. As of version 2.2.0, just use the Presto service type. Presto, also known as PrestoSQL, is the old name for Trino.
+       * The defult policies that Ranger installs under your above service is WIDE open. Everything works for everyone. I will not document the full ranger setup.


We should document the full ranger setup somewhere in Trino land though and point to it.

bitsondatadev · 2022-12-14T11:34:43Z

plugin/trino-ranger/src/main/java/io/trino/plugin/ranger/RangerSystemAccessControlImpl.java

+    @Override
+    public Set<String> filterCatalogs(SystemSecurityContext context, Set<String> catalogs)
+    {
+        LOG.debug("==> RangerSystemAccessControl.filterCatalogs(" + catalogs + ")");


bitsondatadev · 2022-12-14T11:34:54Z

plugin/trino-ranger/src/main/java/io/trino/plugin/ranger/RangerSystemAccessControlImpl.java

+    }
+
+    @Override
+    public void checkCanKillQueryOwnedBy(SystemSecurityContext context, String queryOwner)


bitsondatadev · 2022-12-14T11:35:25Z

plugin/trino-ranger/src/test/java/io/trino/plugin/ranger/RangerSystemAccessControlTest.java

+import static io.trino.spi.security.PrincipalType.USER;
+import static io.trino.spi.security.Privilege.SELECT;
+
+public class RangerSystemAccessControlTest


bitsondatadev · 2022-12-14T11:44:00Z

docs/src/main/sphinx/connector/ranger.rst

@@ -0,0 +1,212 @@
+=======================


I agree, this should likely live under /docs/src/main/sphinx/security in a file called ranger-access-control.rst.

bitsondatadev · 2022-12-14T11:45:31Z

docs/src/main/sphinx/connector.rst

@@ -34,6 +34,7 @@ from different data sources.
    Pinot           <connector/pinot>
    PostgreSQL      <connector/postgresql>
    Prometheus      <connector/prometheus>
+    Ranger          <connector/ranger>


This would move here.

cc: @mosabua

bitsondatadev · 2022-12-14T11:58:11Z

docs/src/main/sphinx/connector/ranger.rst

+
+.. raw:: html
+
+  <img src="../_static/img/apache_ranger.png" class="connector-logo">


We should use the latest Ranger logo. We all love the elephant but anything I see with it now I associate with "outdated and slow".

osscm · 2023-11-02T05:59:19Z

Hey @dprophet
Wondering, is there any update or decision on this PR (adding Ranger plugin to the Trino itself)

I see now its available at Apache Ranger as well (https://github.com/apache/ranger/tree/master/plugin-trino)

mosabua · 2023-11-07T21:59:16Z

In my opinion it would still be great to get a plugin into the Trino codebase. This would ensure that it works against the SPI of the current Trino release. The plugin in the Ranger codebase is using Trino 377 at this stage. This is hopelessly out of date and imposes a lot of work on anyone who wants to use this with Trino since they would have to individually update to their version of Trino.

From what I understand @dprophet is still hoping to pick this up again and continue the work.

lozbrown · 2024-02-05T14:55:24Z

@dprophet any hope on ranger plugin now OPA is merged?

ebyhr · 2024-07-17T00:36:03Z

@dprophet Are you still working on this PR? @mneethiraj sent a new PR #22675

sopel39 · 2024-08-14T16:21:02Z

plugin/trino-ranger/pom.xml

+                    <artifactId>jersey-json</artifactId>
+                </exclusion>
+                <exclusion>
+                    <groupId>com.sun.jersey</groupId>


why is the server needed for ranger client?

github-actions · 2024-09-05T17:03:26Z

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

mosabua · 2024-09-11T23:03:38Z

I think the new PR is a replacement. #22675

Closing this one.

dprophet and others added 3 commits July 20, 2022 09:57

Merge pull request #1 from trinodb/master

90aaf6e

Rebase from Trino master

Merging embedded Apache Ranger plugin.

6c9cb80

Fixing Trino 379->391 merge dependencies.

39e8ed4

Riddle4045 mentioned this pull request Jul 21, 2022

Enable a pub-sub mechanism for cross-plugin, plugin-core communication #13298

Open

Fix Error Prone compile complaint.

a9671e4

cla-bot bot added the cla-signed label Jul 25, 2022

Refactoring Ranger readme.md into standard Trino documentation.

5210b5f

github-actions bot added the docs label Jul 25, 2022

dprophet added 2 commits July 25, 2022 21:42

Added full instructions for Ranger and connecting Trino to Ranger.

5288c22

Fix error-prone compiler complaint.

f3387ab

kokosing reviewed Aug 2, 2022

View reviewed changes

rohank07 mentioned this pull request Aug 22, 2022

Investigate restricting third-party access to user's container StatCan/aaw#1312

Closed

dprophet added 4 commits November 4, 2022 10:43

Fix null error if there is no expressions

b5123ab

forgot collections import

4319e5a

Fix ranger + hadoop dependency versions.

baf4dd8

Exclude org.eclipse.persistence.moxy as a dependency

a1ce26e

bitsondatadev suggested changes Dec 14, 2022

View reviewed changes

bitsondatadev reviewed Dec 14, 2022

View reviewed changes

kokosing force-pushed the master branch from 3f05134 to 58d6356 Compare March 14, 2023 11:34

vagaerg mentioned this pull request Jul 13, 2023

Add support for Open Policy Agent #17940

Closed

5 tasks

mneethiraj mentioned this pull request Jul 18, 2024

Add Apache Ranger authorizer plugin #22675

Open

sopel39 reviewed Aug 14, 2024

View reviewed changes

github-actions bot added the stale label Sep 5, 2024

mosabua closed this Sep 11, 2024


		To connect to Apache Ranger you need:

		* Apache Ranger installed, up and running. Compatible with Ranger v2.1.x, 2.2.x, 2.3.x


		https://ranger.apache.org/

		Apache Ranger is a framework to enable comprehensive data security across many platforms. Trino is one of these platforms.


		### Full Enterprise ready Apache Ranger setup

		* Install and startup Ranger


		.. raw:: html

		<img src="../_static/img/apache_ranger.png" class="connector-logo">

Apache Ranger plugin #13297

Apache Ranger plugin #13297

Conversation

dprophet commented Jul 21, 2022

Description

Related issues, pull requests, and links

Documentation

Release notes

cla-bot bot commented Jul 21, 2022

kokosing left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bitsondatadev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

osscm commented Nov 2, 2023

mosabua commented Nov 7, 2023

lozbrown commented Feb 5, 2024

ebyhr commented Jul 17, 2024

Choose a reason for hiding this comment

github-actions bot commented Sep 5, 2024

mosabua commented Sep 11, 2024