AWS Lambda에 셀레니움, 크롬 드라이버를 사용하는 방법(2)

개발환경

ubuntu 20.04

python 3.8

docker

aws cli

aws sam

aws ecr

aws lambda

selenium

chrome driver

---

이 글을 따라 하려면 알아야 될 기술들이 많다.

https://aaaag.tistory.com/60

ubuntu에 AWS cli 설치

https://aaaag.tistory.com/61

ubuntu에서 AWS sam 설치 및 테스트

어떤 거든간에 첫 시작부터 완벽하게 이해하고 시작할 순 없다.

대충 어느정도 알겠다는 느낌이 오면 바로 시작한 다음 나중에 충분히 공부해서 내 것으로 만들면 되는 거다.

그럼 시작한다.

---

ubuntu에 도커가 설치되어 있어야 한다.

1. 새로운 앱을 만든다

sam init --package-type Image

위 화면과 같이 생성한다.

만약 위와 다르게 필요한 옵션이라면 설정을 다르게 해 주시면 된다.

최종적으로 ls 명령어로 앱이 생성됐는지 확인한다.

나는 mbmove(megabyte move : 메가바이트단위 패킷이동)로 이름을 정했기 때문에 해당 폴더가 생성된 걸 볼 수 있다.

생성된 폴더 구조에서 폴더나 파일의 이름을 변경하자.

• hello_world
  • 폴더 명을 src로 변경
  • app.py파일 내용 변경

## Run selenium and chrome driver to scrape data from cloudbytes.dev
import time
import json
import os.path
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options



def handler(event=None, context=None):
    chrome_options = webdriver.ChromeOptions()
    chrome_options.binary_location = "/opt/chrome/chrome"
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--no-sandbox")
    chrome_options.add_argument("--disable-dev-shm-usage")
    chrome_options.add_argument("--disable-gpu")
    chrome_options.add_argument("--disable-dev-tools")
    chrome_options.add_argument("--no-zygote")
    chrome_options.add_argument("--single-process")
    chrome_options.add_argument("window-size=2560x1440")
    chrome_options.add_argument("--user-data-dir=/tmp/chrome-user-data")
    chrome_options.add_argument("--remote-debugging-port=9222")
    #chrome_options.add_argument("--data-path=/tmp/chrome-user-data")
    #chrome_options.add_argument("--disk-cache-dir=/tmp/chrome-user-data")
    chrome = webdriver.Chrome("/opt/chromedriver", options=chrome_options)
    chrome.get("https://cloudbytes.dev/")
    description = chrome.find_element(By.NAME, "description").get_attribute("content")
    print(description)
    return {
        "statusCode": 200,
        "body": json.dumps(
            {
                "message": description,
            }
        ),
    }

• requirements.txt파일 내용 변경

selenium
requests
pandas

• chrome-deps.txt파일 생성 후 내용 입력

acl adwaita-cursor-theme adwaita-icon-theme alsa-lib at-spi2-atk at-spi2-core 
atk avahi-libs cairo cairo-gobject colord-libs cryptsetup-libs cups-libs dbus 
dbus-libs dconf desktop-file-utils device-mapper device-mapper-libs elfutils-default-yama-scope
elfutils-libs emacs-filesystem fribidi gdk-pixbuf2 glib-networking gnutls graphite2 
gsettings-desktop-schemas gtk-update-icon-cache gtk3 harfbuzz hicolor-icon-theme hwdata jasper-libs
jbigkit-libs json-glib kmod kmod-libs lcms2 libX11 libX11-common libXau libXcomposite libXcursor 
libXdamage libXext libXfixes libXft libXi libXinerama libXrandr libXrender libXtst libXxf86vm libdrm 
libepoxy liberation-fonts liberation-fonts-common liberation-mono-fonts liberation-narrow-fonts 
liberation-sans-fonts liberation-serif-fonts libfdisk libglvnd libglvnd-egl libglvnd-glx libgusb 
libidn libjpeg-turbo libmodman libpciaccess libproxy libsemanage libsmartcols libsoup libthai libtiff 
libusbx libutempter libwayland-client libwayland-cursor libwayland-egl libwayland-server libxcb 
libxkbcommon libxshmfence lz4 mesa-libEGL mesa-libGL mesa-libgbm mesa-libglapi nettle pango pixman 
qrencode-libs rest shadow-utils systemd systemd-libs trousers ustr util-linux vulkan 
vulkan-filesystem wget which xdg-utils xkeyboard-config

• Dockerfile파일 내용 변경

FROM public.ecr.aws/lambda/python:3.9 as stage

# Hack to install chromium dependencies
RUN yum install -y -q sudo unzip

# Find the version of latest stable build of chromium from below
# https://omahaproxy.appspot.com/
# Then follow the instructions here in below URL 
# to download old builds of Chrome/Chromium that are stable
# Current stable version of Chromium
ENV CHROMIUM_VERSION=1002910 


# Install Chromium
COPY install-browser.sh /tmp/
RUN /usr/bin/bash /tmp/install-browser.sh

FROM public.ecr.aws/lambda/python:3.9 as base

COPY chrome-deps.txt /tmp/
RUN yum install -y $(cat /tmp/chrome-deps.txt)

# Install Python dependencies for function
COPY requirements.txt /tmp/
RUN python3 -m pip install --upgrade pip -q
RUN python3 -m pip install -r /tmp/requirements.txt -q 


COPY --from=stage /opt/chrome /opt/chrome
COPY --from=stage /opt/chromedriver /opt/chromedriver
COPY app.py ${LAMBDA_TASK_ROOT}

CMD [ "app.handler" ]

• install-browser.sh파일 생성 후 입력

#!/bin/bash


echo "Downloading Chromium..."
curl "https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/\
Linux_x64%2F$CHROMIUM_VERSION%2Fchrome-linux.zip?generation=1652397748160413&alt=media" > /tmp/chromium.zip

unzip /tmp/chromium.zip -d /tmp/
mv /tmp/chrome-linux/ /opt/chrome

curl "https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/\
Linux_x64%2F$CHROMIUM_VERSION%2Fchromedriver_linux64.zip?generation=1652397753719852&alt=media" > /tmp/chromedriver_linux64.zip

unzip /tmp/chromedriver_linux64.zip -d /tmp/
mv /tmp/chromedriver_linux64/chromedriver /opt/chromedriver

• install-browser.sh파일 권한 부여

chmod +x src/install-browser.sh

• mbmove 폴더 안의 __init__. py파일과 hello_world 폴더안의 __init__.py파일 검사
  • 안에 아무 내용도 있어선 안된다.
• template.yaml 파일 수정

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  python3.8

  Sample SAM Template for mbmove

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 480
    MemorySize: 2048

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      PackageType: Image
      Architectures:
        - x86_64
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /mbmove
            Method: get
    Metadata:
      Dockerfile: Dockerfile
      DockerContext: ./src
      DockerTag: python3.8-v1

Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  MBMoveApi:
    Description: "API Gateway endpoint URL for Prod stage for MBMovie function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/mbmove/"
  HelloWorldFunction:
    Description: "mbmove Lambda Function ARN"
    Value: !GetAtt MBMove.Arn
  MBMoveFunctionIamRole:
    Description: "Implicit IAM Role created for MBMove function"
    Value: !GetAtt MBMoveFunctionRole.Arn

2. 앱을 빌드한다

sam build

Build Succeeded가 출력되면 성공이다.

3. 로컬에서 테스트해본다.

sam local invoke

위 화면처럼 statusCode가 200이면 정상적으로 작동하는 거다.

4. AWS에 개시한다

sam deploy --guided

몇 가지 질문들이 나올 텐데 원하는 내용으로 선택하면 개시된다.

---

참고

도커를 사용하는 sam 앱 생성

https://cloudbytes.dev/snippets/run-aws-lambda-using-custom-docker-container#create-a-new-app

Run AWS Lambda using custom docker container

셀레늄 설치

https://cloudbytes.dev/snippets/run-selenium-in-aws-lambda-for-ui-testing#using-the-github-repository-directly

Run Selenium in AWS Lambda for UI testing