Scala and Spark for Big Data Analytics

Step 4: Creating data and log directories

To run your Spark jobs using Hadoop, it needs to have the data and the log directories with various permissions. You can use the following command:

$  mkdir -p /var/data/hadoop/hdfs/nn$  mkdir -p /var/data/hadoop/hdfs/snn$  mkdir -p /var/data/hadoop/hdfs/dn$  chown hdfs:hadoop /var/data/hadoop/hdfs –R$  mkdir -p /var/log/hadoop/yarn$  chown yarn:hadoop /var/log/hadoop/yarn -R

Now you need to create the log directory where YARN is installed and then set the owner and group as follows:

$  cd /opt/yarn/hadoop-2.7.3$  mkdir logs$  chmod g+w logs$  chown yarn:hadoop . -R