Data Lake for Enterprises

File Data Load

1. Files from a linux machine can be easily copied into HDFS cluster by using fs put command. This command is part of Hadoop client  which can be installed on any Linux machine. In our case, Hadoop client is available as part of Hadoop pseudo-distributed setup.

A general syntax of this command is as given:

hdfs dfs -put /local/path/test.file hdfs://namenode:9000/user/stage

2. For this example, let us create a raw area of data in HDFS (a folder in HDFS). This area would contain the data in its most natural form as acquired from the source using the command:

hdfs dfs -mkdir -p /<any-path>/raw/txt

Once the previous command is executed, it will create the folder structure (<any-path>/raw/txt) in HDFS which can be viewed using ...