AWS Setting guides
Dataverse supports AWS S3 and EMR. If you want to use Dataverse with them, follow this step!
Last updated
Dataverse supports AWS S3 and EMR. If you want to use Dataverse with them, follow this step!
Last updated
SPARK_HOME
is required for the following steps. please make sure you have set SPARK_HOME
before proceeding. You can find setting up SPARK_HOME
guideline in this .
hadoop-aws
& aws-java-sdk
versionhadoop-aws
The version must match with hadoop version. you can check your hadoop version by running below command. while writing this README.md the hadoop version was 3.3.4
so the example will use 3.3.4
version.
aws-java-sdk
The version must be compatible with hadoop-aws version. Check at Maven Compile Dependencies section. (e.g. hadoop-aws 3.3.4 is compatible with aws-java-sdk-bundle 1.12.592)
hadoop-aws
& aws-java-sdk
versionDownload corresponding version of hadoop-aws
and aws-java-sdk
jar files to $SPARK_HOME/jars
directory.
Currently we do not support ENV variables for AWS credentials but this will be supported in the future. Please use aws configure
command to set your AWS credentials and this will set ~/.aws/credentials
file which is accessible by boto3
.
aws_access_key_id
aws_secret_access_key
region
When you have temporary security credentials you have to set session token
too.
now you are ready to use
Dataverse
with AWS! Every other details will be handled byDataverse
!
Makefile can be found on repository of Dataverse [].