Automating processes

Recently I have been working on my AWS Architect certification. Rather than just grinding through the training material and practice exams I thought I should actually build something and journal my process. I have done this internally for Commvault but wanted to do an external blog as well so that if I left Commvault I would have a copy of my notes.

The first step in building something is documenting the project including the goals, objectives, and components that will be needed. Initially I interviewed the person that I am building this project for and drew everything on paper (I know, old school). From there I transcribed it into an AWS architecture diagram using LucidChart.

The goal of the project is to take a zip file that is an aggregation of ECGs performed on student athletes and upload it into S3. Once the zip file is uploaded it kicks off a process that unzips the files and copies them to another folder inside the bucket. From here these files are copies to Dropbox and a notification is sent to one or more email boxes or text messages. From this notification a Cardiologist interprets the results and responds to the email or text. Once the response is received the interpreted files are transferred from Dropbox back into S3 and sorted according to the school that was screened. Students that were marked as low risk are stamped with a low risk label. Students that were marked needing a follow up or high risk are placed into a different folder for manual processing and a notification is sent via email and/or text requesting for manual intervention.

The first step in the process is taking a zip file that was uploaded into S3 and processing it. Fortunately we have the ability to launch processes when a file is uploaded into S3 with the Lambda functions. A good place to start learning about this is https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html where the tutorial talks about how to create a Lambda function and tie it to changes to a specific bucket. This also required creating an IAM role that allows a Lambda function to read and write an S3 bucket.

Step 1) Create IAM Role. This id done by going to the IAM console and selecting Roles at the left of the screen. We want to click on the Create role blue button near the middle of the screen.



We select the Lambda function and click Next:Permissions at the bottom of the screen.

To make things easy we select AmazonS3FullAccess. What we want is the ability to read and write objects in an S3 bucket. We need to type in S3 in the Filter policies then select the box next to AmazonS3FullAccess.

We skip the tags role and go to Review. Here we enter a Role name. We will call it gbgh-processing. Once we enter this information we click Create role at the bottom right of the screen.

At this point we have a role that allows our Lambda function to access S3 objects and manipulate them.

Step 2) The next step is to create a bucket that we will upload files into. This is done by going to the S3 console and creating a bucket. The bucket must be in the same zone that we create our Lambda function. In our example we will create a bucket called gbgh-test and put it in the US East region.

We click on the Create bucket blue button at the top left of the screen. The bucket name needs to be unique and we want the US East region. We don’t want to copy settings from other buckets and will configure the options on our own. We will be using gbgh-test as the bucket name.

We will go with the default options and clear all of the permissions as shown below. We want to open up public access for our bucket because a variety of people long term will be uploading files into the bucket. We can control access through other mechanisms at a later date.

From here we click Next and Create Bucket on the next screen.

We should see our bucket ready and available in the S3 bucket list.

Step 3) create a Lambda function that receives S3 object changes so that we can process the files and do something with it. To do this we go to the Lambda function console.

Notice that we have a few functions already defined. Some are Java. Some are Node.js. Some are Python. We will be creating a Python 2.7 binary so that we can use the boto3 library. This pre-defined library allows us to quickly and easily manipulate objects in a bucket and call other AWS services like email and queue services. To start we click on the orange Create function button at the top right of the screen.

We will Author from scratch our code since we have some simple code that we can work from. We will call the function gbgh-test and select Python 2.7 as the runtime. We will select the gbgh-processing role to give our function access to the S3 bucket that we created.

When we click on the Create function it drops us into the Designer tool for editing and testing our Lambda function. The first thing that we want to do is add an S3 trigger to link S3 objects to our function.

The trigger is found under the Designer – Add triggers list. Scroll down and click on S3. It will show that Configuration is required and we will need to scroll down to configure the trigger. From here we will select the bucket and Event type. We are looking for create events which indicates that someone uploaded a file for us to process. We select the gbgh-test bucket and stick with the All object create events.

After clicking on Add we nativate to the gbgh-test icon to see the Function code where we can edit the function and test code that we want. The default handler does not do what we want so we need to replace this code.

For our function we are going to start simple. We will handle a create event, pull the bucket name from the event handler, pull the file name that was uploaded, and print all of the items in the bucket for testing. The code starts with an import of the boto3 library. We create a handle into S3 with the boto3.resource(‘s3’) library call. The lambda_handler is launched when a file is created in our S3 bucket. From here we print some diagnostics that the handler was called then walk the object list in the bucket and print the object names. From here we return and terminate the Lambda function. Sample code for this can be found at lambda_function_s3_upload.js

The next step is to create a test event to simulate a file upload. We do this by clicking on the Select a test event pulldown and selecting Configure test events.

For the test event we need to simulate an ObjectCreated:Put call or an https PUT that causes a file to be uploaded to our S3 bucket. We need to define the arn for our bucket and bucket name as well as an object name (gbgh.zip) that we need to actually create in our gbgh-test bucket. We define the event name as gbghTest and paste in our simulated event call. The code for this can be found at testS3Put.json

The data at the bottom of the screen are just curly brackets to finish out the record definition. From here we click on Create (I had to scroll down to see it) to create our test function. At this point we can select gbghTest and click on the Test button at the top right of the screen. When we first tried this it failed with an error. We had to add two lines to import the json and urllib functions since we call them to process the keys or object names. Once we make these changes we see that the status came back Succeeded and we get an empty listing of gbgh-test if we scroll down in the Execution results screen.

We can upload our gbgh.zip file into our gbgh-test bucket and it should appear at the bottom of the Execution results screen.

To summarize, we have a Lambda function that gets launched when we upload a file into an S3 bucket. Currently the function just prints a directory of the bucket by listing all of the objects that it contains. We had to create an IAM policy so that our function can interact with S3. In the next blog post we will do some processing of this zip file and send some notifications once the zip file is processed.

Post Views: 4,464

1,531 thoughts on “Automating processes”

  86. I definitely wanted to write down a note to appreciate you for some of the remarkable points you are giving out on this site. My particularly long internet lookup has finally been compensated with sensible information to write about with my co-workers. I ‘d state that that many of us website visitors are undoubtedly endowed to dwell in a good website with so many perfect individuals with insightful strategies. I feel truly fortunate to have come across the web page and look forward to really more fabulous times reading here. Thanks a lot once again for a lot of things.

    Reply

  87. I simply needed to say thanks once more. I am not sure the things that I might have taken care of in the absence of the actual secrets revealed by you relating to this field. It has been an absolute fearsome crisis in my circumstances, nevertheless coming across your specialised way you resolved that took me to jump for joy. I will be happier for the advice and thus sincerely hope you recognize what a great job you happen to be accomplishing educating people all through a web site. Most likely you haven’t met all of us.

    Reply

  94. I would like to show my passion for your kindness for women who actually need help on your area. Your real commitment to getting the message up and down had become definitely valuable and have really empowered girls just like me to reach their pursuits. Your personal insightful advice implies much a person like me and even more to my office colleagues. Many thanks; from each one of us.

    Reply

  95. I and also my pals were found to be reading through the excellent tips and hints located on your web page while immediately I had a terrible suspicion I never expressed respect to the site owner for those tips. The people are already for that reason passionate to study them and now have quite simply been tapping into these things. I appreciate you for really being considerably considerate and also for obtaining this kind of outstanding guides millions of individuals are really needing to discover. My honest regret for not expressing appreciation to you sooner.

    Reply

  108. Needed to compose you the little bit of word to help thank you very much once again on the magnificent ideas you’ve provided on this page. This has been so wonderfully open-handed with you giving easily all some people would have supplied for an electronic book to get some money for themselves, chiefly given that you might well have done it if you ever desired. Those basics as well worked to be the great way to understand that most people have the same dreams just as mine to learn a little more with regards to this problem. I believe there are numerous more fun moments up front for individuals who go through your website.

    Reply

  109. Thank you so much for giving everyone remarkably brilliant opportunity to read articles and blog posts from this website. It can be very good and also packed with a great time for me and my office acquaintances to visit your blog more than three times in one week to see the new items you have got. And definitely, I am also certainly fulfilled concerning the great secrets served by you. Certain 4 facts in this post are in truth the most effective we have ever had.

    Reply

  120. Thanks so much for giving everyone an exceptionally brilliant possiblity to read from this web site. It is usually very superb and as well , jam-packed with fun for me and my office mates to search your website not less than 3 times in one week to see the newest guides you have. Not to mention, we are usually satisfied with all the effective creative concepts you give. Some 1 points in this post are definitely the most impressive we have all ever had.

    Reply

  121. I want to express my admiration for your generosity supporting people that have the need for guidance on the content. Your real commitment to getting the message all around had been definitely functional and has enabled men and women much like me to achieve their aims. Your personal insightful information can mean a great deal to me and especially to my peers. Warm regards; from everyone of us.

    Reply

  243. https://bit.ly/3MBv5p3 , how to change zoom on lenovo laptop , how to change desktop background picture size in windows 7 , how to make your zoom profile default
    how to change the account in zoom , zoom error code 3123 – none: , how do i change the screen size on my chromebook
    how to find video on messenger , zoom login join from browser , how to enlarge desktop background picture
    how do i put zoom on my chromebook , how do i test a zoom connection – how do i test a zoom connection: , zoom video communications stock price today – none:

    https://bit.ly/3NAoPze , why is virtual background not available in zoom – why is virtual background not available in zoom: , how to schedule a meeting on zoom from outlook , how to have whiteboard in zoom
    can you use zoom without a webcam , what is the latest version of zoom 2021 – none: , how to extend a computer monitor – how to extend a computer monitor:
    how many video game genre are there , how to change the password in zoom meeting , how to make my laptop screen rotate – none:

    https://newsfrom965sinquilanepn.blogspot.com/2022/07/can-you-zoom-without-downloading-app.html
    how to insert audio clip into powerpoint , armstrong zoom internet reviews – armstrong zoom internet reviews: , zoom download for windows 10
    how to full screen windows 10 , zoom network connection failed error code 1105 – zoom network connection failed error code 1105: , how to allow access to share screen on zoom

    https://newsfrom4sacficozuy4.blogspot.com/2022/07/how-to-use-zoom-on-acer-laptop.html
    format factory video converter for windows 10 , how to create a zoom meeting registration , how to enlarge computer screen using keyboard – none:
    how to change zoom profile picture in laptop – how to change zoom profile picture in laptop: , how to zoom out apple tv remote , how long does it take for covid rt pcr test results

    https://newsfromcamconraha01.blogspot.com/2022/07/zoom-meeting-free-download-latest.html
    what appears to be the primary message of the video clip , how to increase the computer screen size – none: , is nike stock a buy zacks – none:
    how to zoom a computer screen out – how to zoom a computer screen out: , how to set up a zoom meeting online – none: , zoom stock graph – none:

    https://newsfrom45percepemi77.blogspot.com/2022/06/pcr-test-aanvragen-bergen-op-zoom-geen.html
    sharechat whatsapp video download tamil , how to fix zoomed in screen windows 7 , how to stretch your computer screen – how to stretch your computer screen:
    how long walgreens pcr test results take , how much data does zoom meeting use per hour – how much data does zoom meeting use per hour: , is there something wrong with zoom right now

    https://sites.google.com/view/zggh58msou/olennvtnsh137j9bmjusyk56r
    how do you hyperlink a video in powerpoint , why is zoom time limited – none: , zoominfo license – none:
    camera won’t work on zoom windows 10 – camera won’t work on zoom windows 10: , how to use zoom app without downloading , zoom link not working on pc – none:

    https://sites.google.com/view/18788ugsbh/04fexzop1xir1tfmi2tb5l
    video calling app for windows mobile – video calling app for windows mobile: , how to zoom in in zoom app – how to zoom in in zoom app: , how to zoom in and out your desktop screen
    how to connect to zoom meeting uk , zoom audio not working on dell laptop – zoom audio not working on dell laptop: , zoom stock analysis

    https://quob.net/post-sitemap3.xml
    https://getknowledge.in/post-sitemap7.xml
    https://getknowledge.in/post-sitemap4.xml
    https://power-hosting.nl/
    https://getknowledge.in/post-sitemap3.xml

    Reply

  520. Very good site you have here but I was wanting to know
    if you knew of any message boards that cover the same topics talked about here?
    I’d really love to be a part of community where I can get advice
    from other knowledgeable people that share the same interest.
    If you have any suggestions, please let me know. Bless you!

    Reply

  524. Acetyl hexapeptide-3(argireline) is used in attempts to decrease the visible effects of aging by reducing the deep wrinkles and lines that occur around the forehead and eyes.

    Chemically, when applied as a solution to specific areas of the face, acetyl hexapeptide-3 inhibits the reactions that cause muscles to move or contract – for example when forming facial expressions such as smiling or frowning.

    Reply

  531. I’d like to thank you for the efforts you’ve put in penning this
    blog. I’m hoping to see the same high-grade content by you in the future
    as well. In fact, your creative writing abilities has encouraged me to get
    my own, personal website now 😉

    Reply

  539. Acetyl hexapeptide-3(argireline) is used in attempts to decrease the visible effects of aging by reducing the deep wrinkles and lines that occur around the forehead and eyes.

    Chemically, when applied as a solution to specific areas of the face, acetyl hexapeptide-3 inhibits the reactions that cause muscles to move or contract – for example when forming facial expressions such as smiling or frowning.

    Reply

  553. Have you ever considered publishing an ebook or guest authoring on other blogs?
    I have a blog based on the same ideas you discuss and would
    really like to have you share some stories/information.
    I know my viewers would appreciate your work. If you’re even remotely interested, feel free to send me an e-mail.

    Reply

  554. Greetings I am so happy I found your webpage, I really found you
    by mistake, while I was searching on Digg for something else, Anyhow I am here now and would just like to say cheers for a
    incredible post and a all round enjoyable blog (I also love the theme/design), I don’t have time to browse
    it all at the minute but I have bookmarked it and also included your
    RSS feeds, so when I have time I will be back to read much
    more, Please do keep up the excellent job.

    Reply

  558. Hey I know this is off topic but I was wondering if you
    knew of any widgets I could add to my blog that automatically tweet my newest twitter
    updates. I’ve been looking for a plug-in like this for quite some
    time and was hoping maybe you would have some experience
    with something like this. Please let me know if you run into anything.
    I truly enjoy reading your blog and I look forward to your
    new updates.

    Reply

  583. Amazing blog! Is your theme custom made or did you download it from somewhere?

    A design like yours with a few simple tweeks would really make my blog stand out.
    Please let me know where you got your theme.
    Cheers

    Reply

  584. Have you ever thought about creating an ebook or guest authoring on other sites?
    I have a blog based upon on the same topics you discuss and would really like
    to have you share some stories/information. I know my viewers would value your work.
    If you are even remotely interested, feel free to shoot me an e-mail.

    Reply

  589. Admiring the persistence you put into your site and
    in depth information you provide. It’s great to come
    across a blog every once in a while that isn’t the same out of date rehashed information. Wonderful read!
    I’ve saved your site and I’m adding your RSS feeds to my Google account.

    Reply

  592. Just want to say your article is as astounding.
    The clarity for your submit is just great and that
    i can assume you are knowledgeable on this subject. Fine together with your permission let me to take hold
    of your RSS feed to stay up to date with imminent
    post. Thank you a million and please continue the rewarding work.

    Reply

  599. https://newsfromciacalropu72.blogspot.com/2022/08/kosten-pcr-test-ggd-pcr-test.html
    [url=https://newsfrom8initnenrex0.blogspot.com/2022/08/download-center-zoom.html ] google mail customer service uk [/url] , how do i record zoom meeting , join zoom meeting online from browser – join zoom meeting online from browser:
    [url=https://newsfrom2flatesermuo1.blogspot.com/2022/08/download-zoom-cloud-meetings-for-windows_0673319806.html]download zoom app for free on pc – none: [/url] , check my camera in zoom , can you join two zoom meetings at the same time on different devices – can you join two zoom meetings at the same time on different devices:

    https://newsfromprodamonkevj.blogspot.com/2022/08/zoom-meetings-free-download-64-bit.html
    [url=https://newsfrom472pulcligrahohs.blogspot.com/2022/08/how-do-i-download-zoom-recording-from.html ] video conferencing software best – video conferencing software best: [/url] , how to set photo in zoom app in pc – how to set photo in zoom app in pc: , is zoom a good stock – is zoom a good stock:
    [url=https://newsfrom33macpetcontetw.blogspot.com/2022/08/zoom-meeting-download-laptop.html]add members to zoom channel [/url] , zoom meeting join codes , how do i change the font size on my computer screen windows 7

    https://newsfrom005stulserlaechiz9.blogspot.com/2022/08/download-zoom-cloud-meetings-apk.html
    [url= ] totally free online dating sites over 50 australia [/url] , can i join zoom meeting without password – none: , how do i install zoom on my dell laptop
    [url=]when should rt pcr test be done [/url] , zoom error code 10088 , how to join zoom meeting in browser – how to join zoom meeting in browser:

    https://newsfrom36laeciagedzuqo.blogspot.com/2022/08/zoomusdownload-client-for-meetings-how.html
    [url= ] whatsapp status video download tamil love feeling songs share chat [/url] , hp laptop camera not working on zoom – hp laptop camera not working on zoom: , how to join a meeting on zoom without downloading the app – none:
    [url=]how do i download zoom on my hp laptop [/url] , zoom meeting install location – none: , install zoom for ubuntu 20.04

    https://newsfrom271fleximilwo2l.blogspot.com/2022/08/zoom-download-and-upload-speed.html
    [url=https://newsfromgaesicanu89.blogspot.com/2022/08/download-zoom-for-all-users.html ] what is video conferencing used for – what is video conferencing used for: [/url] , how to set up a zoom link , how to link my zoom account to outlook – how to link my zoom account to outlook:
    [url=https://newsfrom15provitpronsase.blogspot.com/2022/08/zoom-app-for-zoom-phone-appliance.html]how do i allow someone else to share their screen in zoom [/url] , how to save a recurring zoom meeting link , why is zoom computer audio not working – none:

    https://newsfromcietisviogios.blogspot.com/2022/08/login-to-zoom-meeting-with-idrandom.html
    [url=https://newsfrom14deigantosudc.blogspot.com/2022/08/pinch-zoom-jquery-free-download.html ] joy live app 2021 [/url] , zoom stock price history 2019 – none: , how to use zoom mobile app for free video conferencing – none:
    [url=https://newsfrom31spirranahe6q.blogspot.com/2022/08/zoom-555-downloadwere-now-downloading.html]zoom app for pc play store [/url] , why aren’t my speakers working on zoom – none: , how to make my laptop screen zoom out – none:

    https://newsfrom69repcavigogk.blogspot.com/2022/08/download-zoom-app-for-windows-pc-free.html
    [url=https://newsfrom387permulmorse7h.blogspot.com/2022/08/download-free-font-zoom.html ] online free video call app [/url] , how to find meeting passcode on zoom – none: , how do i give someone permission to share their screen on zoom
    [url=https://newsfrom12dirimebowj.blogspot.com/2022/08/one-moment-please.html]zoom error code 404 [/url] , how to contact zoom tech support , how to get in a zoom meeting without host permission – none:

    https://newsfrompropapterzop5.blogspot.com/2022/08/zoom-download-in-laptop.html
    [url=https://newsfromtincnapoput4.blogspot.com/2022/08/om-app-download-all-updated-links.html ] video call download app imo – video call download app imo: [/url] , how to create a zoom meeting via outlook , zoom stock after hours
    [url=https://newsfrom0daiberanaie.blogspot.com/2022/08/mic-not-working-on-zoom-hp-laptop.html]how do i download zoom on my hp laptop [/url] , how to add zoom icon to desktop windows 10 , best speaker for zoom calls – none:

    Reply

  608. Nice post. I used to be checking continuously this weblog and I am inspired!
    Extremely useful information specifically the remaining phase :
    ) I take care of such information much. I used to be seeking this certain info for a very lengthy time.
    Thank you and best of luck.

    Reply

  610. you are really a excellent webmaster. The web site loading pace
    is amazing. It kind of feels that you’re doing any distinctive trick.
    In addition, The contents are masterwork. you have performed a wonderful activity on this topic!

    Reply

  619. unifiram (developmental code name dm-235) is an experimental drug which has antiamnesic effects in animal studies and with significantly higher potency than piracetam. sunifiram is a molecular simplification of unifiram (dm-232). another analogue is sapunifiram (mn-19). as of 2016, sunifiram had not been subjected to toxicology testing, nor to any human clinical trials, and is not approved for use anywhere in the world.

    Reply