setup-spark ✨

This action sets up Apache Spark in your environment for use in GitHub Actions by:

• installing and adding spark-submit and spark-shell to the PATH
• setting required environment variables such as SPARK_HOME, PYSPARK_PYTHON in the workflow

This enables to test applications using a local Spark context in GitHub Actions.

Usage

You will need to setup Python and Java in the job before setting up Spark

Check for the latest Spark versions at https://spark.apache.org/downloads.html

Basic workflow:

steps:
- uses: actions/setup-python@v2
  with:
    python-version: '3.8'
- uses: actions/setup-java@v1
  with:
    java-version: '11'

- uses: vemonet/setup-spark@v1
  with:
    spark-version: '3.2.1'
    hadoop-version: '3.2'

- run: spark-submit --version

See the action.yml file for a complete rundown of the available parameters.

You can also provide a specific URL to download the Spark .tgz and/or a use specific scala version:

- uses: vemonet/setup-spark@v1
  with:
    spark-version: '3.2.1'
    hadoop-version: '3.2'
    scala-version: '2.13'
    spark-url: 'https://archive.apache.org/dist/spark/spark-3.2.1/spark-3.2.1-bin-hadoop3.2-scala2.13.tgz'

Available versions

Check for the latest Spark versions at https://spark.apache.org/downloads.html

The Hadoop version stays quite stable.

The setup-spark action is tested for various versions of Spark and Hadoop in .github/workflows/test-setup-spark.yml

License

The scripts and documentation in this project are released under the MIT License.

Contributions

Contributions are welcome! Feel free to test other Spark versions, and submit issues, or pull requests.

See the contributor's guide for more details.