Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mountCode: True with serverless-python-requirements #92

Open
maranqz opened this issue Jun 22, 2020 · 10 comments
Open

mountCode: True with serverless-python-requirements #92

maranqz opened this issue Jun 22, 2020 · 10 comments

Comments

@maranqz
Copy link

maranqz commented Jun 22, 2020

If mountCode: True deploy crashing other hand mountCode: False deploy is ok.
serverless-python-requirements
serverless-localstack

serverless.yml

Slightly modified https://github.com/serverless/examples/tree/master/aws-python-line-echo-bot

# ⚠️⚠️ REPLACE THIS COMMENT WITH FULL serverless.yml CONTENT
service: aws-python-line-echo-bot
provider:
  name: aws
  runtime: python3.7

functions:
  line_bot:
    handler: handler.webhook
    events:
      - http:
          path: /webhook
          method: POST
plugins:
  - serverless-localstack
  - serverless-python-requirements
custom:
  localstack:
    debug: true
    stages:
      - local
    host: http://localhost  # optional - LocalStack host to connect to
    autostart: true  # optional - start LocalStack in Docker on Serverless deploy
    endpoints:
      # This section is optional - can be used for customizing the target endpoints
      S3: http://localhost:4572
      Lambda: http://localhost:4574
    lambda:
      # Enable this flag to improve performance
      mountCode: True
    docker:
      # Enable this flag to run "docker ..." commands as sudo
      sudo: False
  stages:
    local:
serverless deploy --stage local output
aws-python-line-echo-bot>sls deploy --stage local
Serverless: Load command interactiveCli
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command upgrade
Serverless: Load command uninstall
Serverless: config.options_stage: local
Serverless: serverless.service.custom.stage: undefined
Serverless: serverless.service.provider.stage: dev
Serverless: config.stage: local
Serverless: Using serverless-localstack
Serverless: Reconfiguring service apigateway to use http://localhost:4567
Serverless: Reconfiguring service cloudformation to use http://localhost:4581
Serverless: Reconfiguring service cloudwatch to use http://localhost:4582
Serverless: Reconfiguring service lambda to use http://localhost:4574
Serverless: Reconfiguring service dynamodb to use http://localhost:4569
Serverless: Reconfiguring service kinesis to use http://localhost:4568
Serverless: Reconfiguring service route53 to use http://localhost:4580
Serverless: Reconfiguring service firehose to use http://localhost:4573
Serverless: Reconfiguring service stepfunctions to use http://localhost:4585
Serverless: Reconfiguring service es to use http://localhost:4578
Serverless: Reconfiguring service s3 to use http://localhost:4572
Serverless: Reconfiguring service ses to use http://localhost:4579
Serverless: Reconfiguring service sns to use http://localhost:4575
Serverless: Reconfiguring service sqs to use http://localhost:4576
Serverless: Reconfiguring service sts to use http://localhost:4592
Serverless: Reconfiguring service iam to use http://localhost:4593
Serverless: Reconfiguring service ssm to use http://localhost:4583
Serverless: Reconfiguring service rds to use http://localhost:4594
Serverless: Reconfiguring service ec2 to use http://localhost:4597
Serverless: Reconfiguring service elasticache to use http://localhost:4598
Serverless: Reconfiguring service kms to use http://localhost:4599
Serverless: Reconfiguring service secretsmanager to use http://localhost:4584
Serverless: Reconfiguring service logs to use http://localhost:4586
Serverless: Reconfiguring service cloudwatchlogs to use http://localhost:4586
Serverless: Reconfiguring service iot to use http://localhost:4589
Serverless: Reconfiguring service cognito-idp to use http://localhost:4590
Serverless: Reconfiguring service cognito-identity to use http://localhost:4591
Serverless: Reconfiguring service ecs to use http://localhost:4601
Serverless: Reconfiguring service eks to use http://localhost:4602
Serverless: Reconfiguring service xray to use http://localhost:4603
Serverless: Reconfiguring service appsync to use http://localhost:4605
Serverless: Reconfiguring service cloudfront to use http://localhost:4606
Serverless: Reconfiguring service athena to use http://localhost:4607
Serverless: Reconfiguring service S3 to use http://localhost:4572
Serverless: Reconfiguring service Lambda to use http://localhost:4574
Serverless: Warning: Unable to find plugin named: TypeScriptPlugin
Serverless: Load command deploy
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command requirements:cleanCache
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command generate-event
Serverless: Load command test
Serverless: Load command dashboard
Serverless: Load command output
Serverless: Load command output:get
Serverless: Load command output:list
Serverless: Load command param
Serverless: Load command param:get
Serverless: Load command param:list
Serverless: Load command studio
Serverless: Load command dev
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: config.options_stage: local
Serverless: serverless.service.custom.stage: undefined
Serverless: serverless.service.provider.stage: dev
Serverless: config.stage: local
Serverless: config.options_stage: local
Serverless: serverless.service.custom.stage: undefined
Serverless: serverless.service.provider.stage: dev
Serverless: config.stage: local
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Generated requirements from G:\projects\parse\trash\examples\aws-python-line-echo-bot\requirements.txt in G:\projects\parse\trash\examples\aws-python-line-echo-bot\.ser
verless\requirements.txt...
Serverless: Installing requirements from G:\projects\parse\trash\examples\aws-python-line-echo-bot\.serverless\requirements\requirements.txt ...
Serverless: Running ...
Serverless: Skip plugin function Package.packageService (lambda.mountCode flag is enabled)
Serverless: Injecting required Python packages to package...

  Type Error [ E R R_ I N V A L I D_ A R G_ T Y P E] -----

  TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be one of type string, Buffer, or URL. Received type undefined
      at readFile (fs.js:296:3)
      at go$readFile (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\graceful-fs\graceful-fs.js:118:14)
      at Object.readFile (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\graceful-fs\graceful-fs.js:115:12)
      at Object.readFile (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\universalify\index.js:5:67)
      at injectRequirements (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless-python-requirements\lib\inject.js:23:6)
      at ServerlessPythonRequirements.injectAllRequirements (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless-python-requirements\lib\inject.js:120
:12)
      at ServerlessPythonRequirements.BbPromise.bind.then.then.then (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless-python-requirements\index.js:
183:43)
  From previous event:
      at Object.after [as hook] (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless-python-requirements\index.js:182:10)
  From previous event:
      at PluginManager.invoke (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\lib\classes\PluginManager.js:489:22)
      at PluginManager.spawn (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\lib\classes\PluginManager.js:509:17)
      at BbPromise.try (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\lib\plugins\deploy\deploy.js:122:50)
  From previous event:
      at Object.before:deploy:deploy [as hook] (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\lib\plugins\deploy\deploy.js:102:22)
      at BbPromise.reduce (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\lib\classes\PluginManager.js:489:55)
  From previous event:
      at PluginManager.invoke (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\lib\classes\PluginManager.js:489:22)
      at getHooks.reduce.then (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\lib\classes\PluginManager.js:524:24)
  From previous event:
      at PluginManager.run (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\lib\classes\PluginManager.js:524:8)
      at variables.populateService.then (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\lib\Serverless.js:131:33)
  From previous event:
      at Serverless.run (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\lib\Serverless.js:118:74)
      at serverless.init.then (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\bin\serverless.js:80:26)
      at runCallback (timers.js:705:18)
      at tryOnImmediate (timers.js:676:5)
      at processImmediate (timers.js:658:5)
      at process.topLevelDomainCallback (domain.js:126:23)
  From previous event:
      at Object.<anonymous> (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\bin\serverless.js:80:4)
      at Module._compile (internal/modules/cjs/loader.js:778:30)
      at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
      at Module.load (internal/modules/cjs/loader.js:653:32)
      at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
      at Function.Module._load (internal/modules/cjs/loader.js:585:3)
      at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
      at startup (internal/bootstrap/node.js:283:19)
      at bootstrapNodeJSCore (internal/bootstrap/node.js:622:3)

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information ---------------------------
     Operating System:          win32
     Node Version:              10.16.3
     Framework Version:         1.73.1
     Plugin Version:            3.6.13
     SDK Version:               2.3.1
     Components Version:        2.31.0

┆Issue is synchronized with this Jira Task by Unito

@maranqz
Copy link
Author

maranqz commented Jun 27, 2020

Problem with serverless-python-requirements, it's trying add requirements to unexist artifacte.
For resolve it by hack need to use

pythonRequirements:
  layer: true
# or
pythonRequirements:
  zip: true

For resolving by serverless-localstack need to extend skipIfMountLambda for override private js function like that.
https://github.com/UnitedIncome/serverless-python-requirements/blob/master/index.js#L166

@whummer
Copy link
Member

whummer commented Jul 29, 2020

Hi @maranqz , thanks for reporting. Good catch about the js function that would have to be overwritten. Do you think you could try and help create a pull request to add this missing functionality? That would be really appreciated! Thanks for your help.

@whummer whummer added bug Something isn't working configuration labels Jul 29, 2020
@chris-erickson
Copy link

Any movement on this? It still doesn't seem possible to mount python code into Localstack? The above options yield layer issues for me, even doing all the serverless-python-requirements don't seem to help either:

    pythonRequirements:
        usePoetry: true
        dockerizePip: "non-linux"
        layer: true
    localstack:
        stages:
            - local
        host: http://localhost
        edgePort: 4566
        autostart: false
        lambda:
            mountCode: true
        docker:
            sudo: false
        debug: true
functions:
    hello:
        handler: app.lambda_handler
        ...
        layers:
            - Ref: PythonRequirementsLambdaLayer

Is it possible this is a limitation of the not-paid version of Localstack?

@srinivaskalyani
Copy link

Hi @whummer , gentle bump on this again. This problem still persists. I am still unable to mount the lambda code.

@whummer
Copy link
Member

whummer commented Nov 11, 2021

Hi @chris-erickson @srinivaskalyani, thanks for the update, and apologies for the delay. In addition to the serverless.yml, could you please share a copy of your startup configs (e.g., docker-compose.yml) with us, so we can fully reproduce this use case end-to-end? Thanks for your help!

@srinivaskalyani
Copy link

srinivaskalyani commented Nov 11, 2021

Hi @whummer ,
Thanks a lot for taking this up again and sorry for being a bother. I am using a local installation of Localstack, not the Docker container. I am sharing a copy of my serverless.yml. Please let me know if I must provide any further information.

service: thetenant-service
plugins:
  - serverless-python-requirements
  - serverless-pseudo-parameters
  - serverless-localstack

custom:
  pythonRequirements:
    dockerizePip: false
    layer: true
  localstack:
    stages:
      - local
    debug: true
    host: http://localhost
    edgePort: 4566
    region: us-east-1
    lambda:
      mountCode: true

package:
  exclude:
    - ./**
  include:
    - my_serverless_root/**

provider:  
  name: aws
  runstime: python3.7
  stage: ${opt:stage, 'default_stage'}
  tenant: thetenant
  region: us-east-1
  deploymentBucket: my-deployment-bucket
  role: arn:aws:iam::#{AWS::AccountId}:role/MyLambdaExecutionRole
  environment:
    STAGE: ${self:provider.stage}
    TENANT: ${self:provider.tenant}

functions:
  my_function:
    name: ${self:provider.tenant}-my-function
    handler: my_handler_dir.aws.my_handler_file.handler
    events:
      - http: POST /my_api/endpoint

@philihp
Copy link

philihp commented Jan 12, 2022

Bump

@ptrhck
Copy link

ptrhck commented Jun 6, 2022

Assuming that this is resolved and you can mount your code, I am still wondering if you will be able to run this code. Because, the idea of using serverless-python-requirements is to add some third-party Python libraries the code depends on. With mountCode: true only the source code will be mounted, packaging is skipped and, thus, no third-party libraries will be available at runtime of the lambda function.

Am I missing the part where third-party dependencies are considered when using local code mounting? This issue also points to that problem localstack/localstack#6181

Can the packaging be forced, even with mounCode?

@steffyP
Copy link
Member

steffyP commented Oct 13, 2023

Hi @ptrhck,

to best of my knowledge, the serverless-python-requirements is currently not considered, e.g. if you use the mountCode: true all dependencies should be in the directory that will be mounted.

@steffyP steffyP added status: backlog and removed bug Something isn't working configuration labels Oct 13, 2023
@barocsi
Copy link

barocsi commented Jan 14, 2025

I'm running into the issue of being able to use mountcode with python, the lambda module is properly found, but dependencites are not. Any ideas what can be added?

package:
individually: true
include:
- paths added already

plugins:

  • serverless-python-requirements
  • serverless-localstack

localstack:
host: http://localhost
edgePort: 4566
autostart: true
lambda:
mountCode: ../../

yet for modules that should be loaded from the virtual environment I get load error
If mountcode is not turned on it's working properly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

8 participants