0

I have a cronjob to run a script at startup, which i created using crontab -e

@reboot /home/ubuntu/startup.sh > /home/ubuntu/log.log 2>&1

My startup.sh script contains:

#!/bin/bash

...
bash /home/ubuntu/ec2-script.sh

and my ec2-script.sh contains:

spark-submit

But i'm getting spark-submit: command not found. If I try from terminal, it is working perfectly fine. But when the script runs at boot up, it is unable to find spark-submit command. I try to put sleep as well, so the spark starts properly, but it doesn't help. It would be great help if someone points out what is wrong or missing.

damadam
  • 2,815
  • 3
  • 17
  • 38
Waqar Ahmed
  • 101
  • 1
  • 2
  • Have you tried to use full path for spark-submit ? – Soren A Aug 09 '18 at 10:29
  • Yeah, It worked with full path. Can I ask why? – Waqar Ahmed Aug 09 '18 at 10:31
  • 1
    It works with full path, because cron don't set up the environment as you have in terminal / bash. This means, among other things, that PATH isn't set. If it works please accept my answer below. – Soren A Aug 09 '18 at 10:38
  • 2
    Possible duplicate of [Why crontab scripts are not working?](https://askubuntu.com/questions/23009/why-crontab-scripts-are-not-working) – David Foerster Aug 09 '18 at 16:09

1 Answers1

2

You will have to use full path for spark-submit.

Cron don't set up the environment as you have in terminal / bash. This means, among other things, that PATH (search path for executables) isn't set.

You can set the environment either in the first lines of crontab, like:

PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
OTHERVAR=/some/thing

or in your script. If you set them in the script you will have to export PATH and other variables you set, to get them passed down when you call other scripts or programs.

Soren A
  • 6,442
  • 2
  • 17
  • 33