Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-1542

PutS3Object Processor Logs Errors Without ListBucketMultipartUploads Permission

Agile BoardAttach filesAttach ScreenshotVotersStop watchingWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 0.5.0
    • 0.5.1
    • Extensions
    • None
    • Linux, Chrome

    Description

      In 0.5.0, I am seeing an error putting objects to Amazon S3 using the PutS3Object processor. The put operation itself succeeds, but a nasty-gram is displayed in the flow and logged as a side effect. The error text is "Error checking S3 Multipart Upload list for <my s3 bucket>", from line 698 of PutS3Object.java. This code was introduced in NIFI-1107, expanding the PutS3Object processor with support for S3 multi-part uploads of larger files.

      Logged in nifi-app.log:

      2016-02-19 20:56:24,312 ERROR [Timer-Driven Process Thread-3] o.a.nifi.processors.aws.s3.PutS3Object PutS
      3Object[id=47df7533-91b2-4635-b865-45323f73f6c3] Error checking S3 Multipart Upload list for batchiq-scra
      tch: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 7483E76EA
      52E33C4)
      
      2016-02-19 20:56:24,533 INFO [Timer-Driven Process Thread-3] o.a.nifi.processors.aws.s3.PutS3Object PutS3
      Object[id=47df7533-91b2-4635-b865-45323f73f6c3] Successfully put StandardFlowFileRecord[uuid=a4cc19f5-cd2
      4-4be7-8157-21f2609fa8cd,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1455912275117-1, container=default, section=1], offset=3072, length=1024],offset=0,name=273670477119454,size=1024] to Amazon S3 in 327 milliseconds
      

      The error-but-success behavior appears to happen because the exception is caught, logged, but not re-thrown. The approximate code path is as follows:

      1. PutS3Object::onTrigger method calls ageoffS3Uploads regardless of single-part/multi-part code path
      2. ageoffS3Uploads calls getS3AgeoffListAndAgeoffLocalState
      3. getS3AgeoffListAndAgeoffLocalState calls s3.listMultipartUploads
      4. After the exception, onTrigger proceeds with the single-part upload path

      Troubleshooting Notes
      Listing multi-part S3 uploads requires an additional S3 permission, s3:ListBucketMultipartUploads. I would expect the PutS3Object processor to only require the s3:PutObject permission for single-part uploads.

      I tested that this behavior did not occur in NiFi 0.4.2. I also tested that adding the s3:ListBucketMultipartUploads permission to my AWS credential circumvents the error messages from the S3PutObject processor.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jskora Joe Skora
            jameswing James Wing
            Votes:
            0 Vote for this issue
            Watchers:
            5 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment