Cannot create triggers for SNS events

[parroyo]

There is an error running zappa schedule when SNS event sources are configured:

$ zappa schedule <stage>
Scheduling..
ERROR:Unable to add SNS event source
Traceback (most recent call last):
  File "/av_home/venv/local/lib/python2.7/site-packages/kappa/event_source/sns.py", line 50, in add
    Endpoint=function.arn)
  File "/av_home/venv/local/lib/python2.7/site-packages/kappa/awsclient.py", line 86, in call
    data = op(**kwargs)
  File "/av_home/venv/local/lib/python2.7/site-packages/botocore/client.py", line 278, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/av_home/venv/local/lib/python2.7/site-packages/botocore/client.py", line 572, in _make_api_call
    raise ClientError(parsed_response, operation_name)
ClientError: An error occurred (InvalidParameter) when calling the Subscribe operation: Invalid parameter: Lambda endpoint ARN
Created sns:Publish event schedule for <module>.<function>!

In zappa_settings.json:

"events": [
            {
                "function": "<module>.<function>",
                "event_source": {
                    "arn":  "<sns arn>",
                    "events": [
                        "sns:Publish"
                    ]
                }
            }
        ]

Looking at the code it seems that the problem is how the arn is split and used in Kappa event source:

In S3EventSource.add:
'LambdaFunctionArn': '%s:%s' % (function.arn, function._context.environment),

But in SNSEventSource.add:
Endpoint=function.arn)

As the value of 'arn' is split in order to work with the S3EventSource it doesn't work fine with SNS cause it expects the full arn value.

    split_arn = lambda_arn.split(':')
    arn_front = ':'.join(split_arn[:-1])
    arn_back = split_arn[-1]
    ctx.environment = arn_back

    ....
    funk.arn = arn_front
    funk._context = ctx

I don't know if it should be fixed in Kappa or in Zappa. For me is strange the way Kappa is building the 'LamdbaFuncionArn' in S3EventSource using the context.environment...

[Miserlou]

Damn, I was afraid of this.

Yeah, this is directly related to: garnaat/kappa#81

I'd rather have this fixed in Kappa, what we're doing is pretty dirty already, and I'd rather not have to check the the event type. Thanks for digging into this one and surfacing this.

Not sure how to proceed. It basically means getting those four event source types patched in Kappa, getting a PR merged and then having them update the deployed version with a Zappa version bump in tandem.

[Miserlou]

Somebody has already submitted a reasonable PR for this, hopefully it'll land upstream soon.

[Miserlou]

Pushed a temporary workaround to master.

[parroyo]

@Miserlou I've just test it, and the deployment works fine using master branch.

But I seems that the Sns events are not processed correctly by the lambda handler.
Looking at the handler code:

        elif event.get('Records', None):

            records = event.get('Records')
            event_types = ['dynamodb', 'kinesis', 's3', 'sns', 'events']

            for record in records:
                for event_type in event_types:
                    if record.has_key(event_type):

                        whole_function = record[event_type]['configurationId']
                        app_function = self.import_module_and_get_function(whole_function)
                        result = self.run_function(app_function, event, context)
                        print(result)

            return result

The 'configurationId' is a specific property of the s3 events. The Sns events don't have it. I guess this should be more generic in order to work with other trigger sources, or handling each event type in a particular way.

[Miserlou]

Yeah, very big + 1.

Related: http://docs.aws.amazon.com/lambda/latest/dg/eventsources.html

[Miserlou]

Related: garnaat/kappa#82