Skip to content

dataherb.utils.awscli

utils.awscli¤

aws_cli(cmd) ¤

aws_cli invokes the aws cli processes in python to execute awscli commands.

Warning

This is not the most elegant way of using awscli.

However, it has been a convinient function in data science projects.

Examples

AWS credential env variables should be configured before calling this function. The awscli command should be wrapped as a tuple. To download data from S3 to a local path, use

>>> aws_cli(('s3', 'sync', 's3://s2-fpd/augmentation/', '/tmp/test'))
Similarly, upload is done in the following way
>>> # local_path = ''
>>> # remote_path = ''
>>> _aws_cli(('s3', 'sync', local_path, remote_path))

References

This function is adapted from https://github.com/boto/boto3/issues/358#issuecomment-372086466

Parameters:

Name Type Description Default
*cmd tuple

tuple of awscli command.

 ()
Source code in dataherb/utils/awscli.py 
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
def aws_cli(*cmd):
    """
    aws_cli invokes the aws cli processes in python to execute awscli commands.

    !!! warning
        This is not the most elegant way of using awscli.

        However, it has been a convinient function in data science projects.


    !!! note "Examples"

        AWS credential env variables should be configured before calling this function.
        The awscli command should be wrapped as a tuple. To download data from S3 to a local path, use

        ```python
        >>> aws_cli(('s3', 'sync', 's3://s2-fpd/augmentation/', '/tmp/test'))
        Similarly, upload is done in the following way
        >>> # local_path = ''
        >>> # remote_path = ''
        >>> _aws_cli(('s3', 'sync', local_path, remote_path))
        ```

    !!! note "References"

        This function is adapted from https://github.com/boto/boto3/issues/358#issuecomment-372086466


    :param tuple *cmd: tuple of awscli command.
    """
    old_env = dict(os.environ)
    try:
        # Set up environment
        env = os.environ.copy()
        env["LC_CTYPE"] = "en_US.UTF"
        os.environ.update(env)

        # Run awscli in the same process
        exit_code = create_clidriver().main(*cmd)

        # Deal with problems
        if exit_code > 0:
            raise RuntimeError(f"AWS CLI exited with code {exit_code}")
    finally:
        os.environ.clear()
        os.environ.update(old_env)