[Suggestion] Switch the need to have the aws cli to using the boto3 python library.

[Aiden-RC]

Using an unsigned boto3 client you can download the required zip file without the need to have the aws cli installed.
Following code below works and is completely python based ( makes the project more multi OS friendly aswell ).

import os
import boto3
from botocore.exceptions import ClientError
from botocore import UNSIGNED
from botocore.config import Config
import zipfile

PUBLIC_DATA_FOLDER_NAME = 'ojd_daps_skills_data'
PROJECT_DIR = os.getcwd()

def download():

    aws_region = 'eu-west-1'
    s3 = boto3.client('s3', region_name=aws_region, config=Config(signature_version=UNSIGNED))

    bucket_name = 'open-jobs-indicators'
    key = f'escoe_extension/{PUBLIC_DATA_FOLDER_NAME}.zip'
    public_data_dir = os.path.join(PROJECT_DIR, PUBLIC_DATA_FOLDER_NAME)

    try:
        s3.download_file(bucket_name, key, f'ojd_daps_skills_data.zip')

        with zipfile.ZipFile(f'{public_data_dir}.zip', 'r') as zip_ref:
            zip_ref.extractall(PROJECT_DIR)

        os.remove(f'{public_data_dir}.zip')

    except ClientError as e:
        print(f"Error: {e}")

download()

[sqr00t]

Heya @Aiden-RC , apologies for the delayed response. Since Cython 3 release in March broke PyYAML which is depended upon by awscli and some other packages, your suggestion has been the general idea in the case of awscli.

If you'd like, we're keen to add your contribution to the project tree. Would you like to open a PR addressing this issue, or be tagged where we will include your code snippet?

[Aiden-RC]

Hiya @sqr00t, i'll open a PR sometime towards the end of the week with the integrated snippet.

[sqr00t]

Heya, I hope you haven't started any extra work yet. There's been some internal demand to fix awscli dependency to get the package working asap. So apologies, I've already adapted your snippet into the above linked PR.

We really appreciate your contribution, and will eventually mention in docs/ elsewhere soon. In the mean time feel free to test out the range of quick fixes by adding the following to your requirements.txt:

ojd-daps-skills @ git+https://github.com/nestauk/ojd_daps_skills.git@201_update_dependencies

I'm keen to get your input on whether the above works for you and/ or if you run into any other related issues :)

[Aiden-RC]

Heya, I hope you haven't started any extra work yet. There's been some internal demand to fix awscli dependency to get the package working asap. So apologies, I've already adapted your snippet into the above linked PR.

We really appreciate your contribution, and will eventually mention in docs/ elsewhere soon. In the mean time feel free to test out the range of quick fixes by adding the following to your requirements.txt:

ojd-daps-skills @ git+https://github.com/nestauk/ojd_daps_skills.git@201_update_dependencies

I'm keen to get your input on whether the above works for you and/ or if you run into any other related issues :)

Ahh Awesome, i hadn't started any work yet so thats not a problem.
I'll test that PR out tomorrow, i'm running it on windows for testing so is a whole kettle of fish as needs the windows
related changes i made previously ( will need updating as i had also tweaked the download function to use the zip file stuff on windows.

Thanks for adapting my snipped 👍

[sqr00t]

Just out of interest, as I may have missed this elsewhere, which Windows version are you using and is WSL part of your stack? I might just quickly add a Windows workflow for CI/CD.

edit: do you use bash, powershell, or cmd as your terminal choice?