Data Lake for Enterprises

Initial Cleanup

Drop the tables from the Hive metadata with commands as shown here in Hue UI. Drop any other additional tables if present:

drop table customer;drop table address;drop table contacts;

Stop the dfs service (stop-dfs.sh) and clean up the Hadoop storage by formatting the Hadoop NameNode with the following command:

${HADOOP_HOME}/bin/hdfs namenode -format

Create new Hadoop directories with the following commands:

hdfs dfs -mkdir -p /datalake/raw/customerhdfs dfs -mkdir -p /datalake/raw/addresshdfs dfs -mkdir -p /datalake/raw/contact

Remove the topics from Kafka servers to ensure we start clean, by using the following commands:

1. Get a list of all the topics:

${KAFKA_HOME}/bin/kafka-topics.sh --list --zookeeper 0.0.0.0:2181 ...