Setting PYTHONPATH breaks the Python Runtime's logic for finding dependencies

[forbushbl]

Describe the bug
The otel-instrument wrapper sets the PYTHONPATH environment variable (see https://github.com/open-telemetry/opentelemetry-lambda/blob/main/python/src/otel/otel_sdk/otel-instrument#L79) but this breaks logic in bootstrap.py which is part of the Python runtime.

Steps to reproduce

1. Create a lambda layer with any library in the python/lib/python3.x/site-packages directory (See documentation)
2. Create a lambda function which depends on the library
3. In the AWS Console open your lambda
4. Go to Configuration > Monitoring and operations tools > Additional monitoring tools > Edit > Enable Application Signals
5. Run your lambda function

What did you expect to see?
The Lambda should run normally and telemetry should be sent to Application Signals

What did you see instead?
There is a Runtime.ImportModuleError in the logs and the Lambda fails to run

What version of collector/language SDK version did you use?
Version: arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-python-arm64-ver-1-29-0:1

What language layer did you use?
Python

Additional context
In bootstrap.py (not posted here due to licensing restrictions), there is logic which checks to see if PYTHONPATH is set. If PYTHONPATH is not set, bootstrap.py will automatically set Python to look in both the /opt/python and /opt/python/lib/python3.x/site-packages directories. It is common for users to place their dependencies in the /opt/python/lib/python3.x/site-packages directory since this is the approach described in AWS's documentation. However, since otel-instrument sets PYTHONPATH and only sets it to include the /opt/python path, any lambda functions which have their dependencies in the /opt/python/lib/python3.x/site-packages will experience an error like:

[ERROR] Runtime.ImportModuleError: Unable to import module 'otel_wrapper': No module named 'oauthlib'

I've documented a workaround in #254 but this isn't ideal because it doesn't address other issues that would be handled by letting bootstrap.py build the path itself.

Recommended fix
Place all of opentelemetry-lambda's dependencies in the /opt/python directory so that you don't need to set the PYTHONPATH.

[serkan-ozal]

@forbushbl What if wrapper script also adds/appends /opt/python/lib/python3.x/site-packages (x will be replaced by actual Python minor version number) path to the PYTHONPATH here (along with /opt/python as is): https://github.com/open-telemetry/opentelemetry-lambda/blob/main/python/src/otel/otel_sdk/otel-instrument#L79?

[forbushbl]

@serkan-ozal Yes, I believe that approach could could work. Although the challenge may be figuring out what value should be used for x in python3.x. The way that bootstrap.py does this is by getting sys.version_info.major and sys.version_info.minor from the running Python interpreter.

I've been thinking you may be able to simplify this by updating the path within otel-wrapper.py. You would add /opt/python and any other paths you need using sys.path.insert(0, '/opt/python') before the import statements in the script. If this works, this would probably allow you to avoid setting PYTHONPATH altogether and you could remove all of these lines from otel-instrument script.

If you could avoid setting PYTHONPATH, you could ensure that you wouldn't break any of the other undocumented behaviors of bootstrap.py.