Errors:
Hadoop Common Errors with Possible Solution
Here I’m writing some of the Hadoop Issue faced by me and providing the solution with it hope you all get the benefit from it.
Hadoop cluster namenode formatted (bin / hadoop namenode -format)
restart cluster will appear as follows
1. After formatted name node (bin/hadoop namenode - format), Its come up with namespace Error,
ERROR: Incompatible namespaceIDS in ...: namenode namespaceID = ...,
datanode namespaceID = ...
Error because the format namenode will re-create a new namespaceID, so
that the original and datanode inconsistent.
Solution:
1. Data files deleted the datanode dfs.data.dir directory (default is tmp /
dfs / data)
2. Modify dfs.data.dir / current / VERSION file the namespaceID and
namenode identical to (log errors where there will be prompt)
3. To reassign new dfs.data.dir directory
2. Hadoop cluster is started with start-all.sh, slave always fail to start datanode, and will get an error:
ERROR: Could only be replicated to 0 nodes, instead of 1
Is the node identification may be repeated (personally think the wrong reasons). There may also be other reasons, and what solution then tries to solve.
Solution:
1. If port access, you should make sure the port is open, such as hdfs :/
/ machine1: 9000 / 50030,50070 like. Executive # iptables-I INPUT-p tcp-dport 9000-j ACCEPT command. If there is an error:
hdfs.DFSClient: Exception in createBlockOutputStream
java.net.ConnectException: Connection refused in; datanode port can not access, modify iptables: # iptables-I INPUT-s machine1-p tcp-j datanode on ACCEPT
2. There may be firewall restrictions between clusters to communicate with each other. Try to turn off the firewall. / Etc / init.d / iptables stop
3. Finally, there may be not enough disk space, check df -al
3. The program execution
Error: java.lang.NullPointerException
Null pointer exception, to ensure that the correct java program. Instantiated before the use of the variable what statement do not like array out of bounds. Inspection procedures.
When the implementation of the program, (various) error, make sure that the situation:
1. Premise of your program is correct by compiled
2. Cluster mode, the data to be processed wrote HDFS path and ensure correct
3. Specify the execution of jar package the entrance class name (I do not know why sometimes you do not specify also can run)
The correct wording similar to this:
$ hadoop jar myCount.jar myCount input output
4. Hadoop start datanode
ERROR: Unrecognized option:-jvm Could not the create the Java virtual machine.
Hadoop installation directory / bin / hadoop following piece of shell:
CLASS = 'org.apache.hadoop.hdfs.server.datanode.DataNode'
if [[$ EUID-eq 0]]; then
HADOOP_OPTS = "$ HADOOP_OPTS-jvm server $
HADOOP_DATANODE_OPTS"
else
HADOOP_OPTS = "$ HADOOP_OPTS-server $
HADOOP_DATANODE_OPTS"
if
$ EUID user ID, if it is the root of this identification will be 0, so try not to
use the root user to operate hadoop .
5.Terminal error message:
ERROR hdfs.DFSClient: Exception closing file / user / hadoop /
musicdata.txt: java.io.IOException: All datanodes 10.210.70.82:50010 are bad. Aborting ...
There are the jobtracker logs the error information
Error register getProtocolVersion
java.lang.IllegalArgumentException: Duplicate metricsName:
getProtocolVersion
And possible warning information:
WARN hdfs.DFSClient: DataStreamer Exception: java.io.IOException: Broken pipe
WARN hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_3136320110992216802_1063java.io.IOException: Connection reset by peer
WARN hdfs.DFSClient: Error Recovery for block
blk_3136320110992216802_1063 bad datanode [0] 10.210.70.82:50010
put: All datanodes 10.210.70.82:50010 are bad. Aborting ...
The solution:
1. Path of under the dfs.data.dir properties of whether the disk is full, try hadoop fs -put data if the processing is full again.
2. Related disk is not full, you need to troubleshoot related disk has no bad sectors, need to be detected.
6.Hadoop jar program get the error message:
java.io.IOException: Type mismatch in key from map: expected
org.apache.hadoop.io.NullWritable, recieved
org.apache.hadoop.io.LongWritable
Or something like this:
Status: FAILED java.lang.ClassCastException:
org.apache.hadoop.io.LongWritable cannot be cast to
org.apache.hadoop.io.Text
Then you need to learn the basics of Hadoop and map reduce model. In
"hadoop Definitive Guide book” in Chapter Hadoop I / O and in Chapter VII,
MapReduce type and format. If you are eager to solve this problem, I can also tell you a quick solution, but this is bound to affect you later development:
Ensure consistent data:
... Extends Mapper ...
public void map (k1 k, v1 v, OutputCollector output) ...
...
... Extends Reducer ...
public void reduce (k2 k, v2 v, OutputCollector output) ...
...
job.setMapOutputKeyClass (k2.class);
job.setMapOutputValueClass (k2.class);
job.setOutputKeyClass (k3.class);
job.setOutputValueClass (v3.class);
...
Note that the corresponding k * and v *. Recommendations or two chapters
I just said. Know the details of its principles.
7. If you hit a datanode error as follows:
ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
java.io.IOException: Cannot lock storage / data1/hadoop_data. The directory is already locked.
According to the error prompts view, it is the directory locked, unable to read. At this time you need to look at whether there are related process is still running or slave machine hadoop process is still running, use the linux
command to view:
Netstat -nap
ps-aux | grep Related PID
If hadoop related process is still running, use the kill command to kill can.
And then re-use start-all.sh.
8. If you encounter the jobtracker error follows:
ERROR: Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailingout.
Solution, modify datanode node /etc/hosts file.
Hosts under brief format:
Each line is divided into three parts: the first part of the network IP address, the second part of the host name or domain name, the third part of the host alias detailed steps are as follows:
1.first check the host name:
$ echo –e “ `hostname - i ` \t `hostname -n` \t $stn ”
Stn= short name or alies of hostname.
It will result in something like that
10.200.187.77 hadoop-datanode DN
If the IP address is configured on successfully modified, or show host name there is a problem, continue to modify the hosts file,
The shuffle error still appears this problem, then try to modify the configuration file of another user said hdfs-site.xml file, add the following:
dfs.http.address
*. *. *: 50070 The ports do not change, instead of the asterisk IP hadoop information transfer through HTTP, the port is same.
9.If you encounter the jobtracker error follows:
ERROR: java.lang.RuntimeException: PipeMapRed.waitOutputThreads ():
subprocess failed with code *
This is a java throws the system returns an error code, the meaning of the error code indicates details.
Please Excuse my typos and please share comment if you feel anything i left out.
10. If you encounter the following error:
FAILED java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI: ***
URI inside the characters not allowed, such as the colon: the class, the operating system does not allow the file named characters. In detail according to the prompt part (part asterisk) , Elimination of illegal character
can solve this issue.
11. To encounter tasktracker not start, tasktracker log error as follows:
ERROR org.apache.hadoop.mapred.TaskTracker: Cannot start task tracker because java.net.BindException: Address already in use ***
Ports are occupied or have corresponding process starts when you first stop the cluster, and then use ps-aux | grep hadoop command to look at the related process of hadoop , And Kill hadoop daemon .
12. Encounter datanode not started, the datanode log error is as follows:
ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
java.io.IOException: No locks available
No locks available can mean that you are trying to use hadoop on a filesystem that does not support file level locking.\
13. Are you trying to run your name node storage in NFS space?
Mentioned file-level locking, use
$ / Etc / init.d / nfs status
Command to view network file system, are closed. Another command df-Th
or mount the type of the file system, you can view the results obtained is indeed the NFS file system. Hanging in the network file system can not be used, because seemingly read-only, if not read-only situation, like said
above, does not support the file-level locking.
Finally the solution, you can try to add a file-level locking to nfs. I am here is to modify the dfs.data.dir, not to use nfs bin.
You can also try to format you Hadoop cluster (if it’s new one) and start all over again.
14. The datanode died, and you cannot start the process, log reported the following errors:
2012-06-04 10:31:34,915 INFO
org.apache.hadoop.hdfs.server.common.Storage: Cannot access storage
directory / data5/hadoop_data
2012-06-04 10:31:34,915 INFO
org.apache.hadoop.hdfs.server.common.Storage: Storage directory /data5/hadoop_data does not exist.
2012-06-04 10:31:35,033 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: org.apache.hadoop.util.DiskChecker $ DiskErrorException: Invalid value for volsFailed: 2, Volumes tolerated: 0
This problem, I found reason the nodes of the disk becomes read-only (only read) mode, online search, I found that quite a number of this case, Linux machine's hard disk could have been set to read-write (Read / Write) mode, but occasionally found automatically becomes read-only (Read Only), check some information, this may happens for various reasons, may be the problem:
· File system errors
· Kernel hardware driver bug
· The FW firmware classes problem
· Disk bad sectors
· Hard disk backplane fault
· Hard drive cable fault
· HBA card failure
· RAID card failure
· inode resource depletion
The solution:
· Restart the server (command reboot)
· Re-mount the hard disk
· fsck try to repair
· Replace the hard disk
15. When running mapreduce task error is as follows:
2012-06-21 10:50:43,290 WARN org.mortbay.log: / mapOutput:
org.apache.hadoop.util.DiskChecker $ DiskErrorException: Could not find
taskTracker/hadoop/jobcache/job_201206191809_0004/attempt_20120619
1809_0004_m_000006_0/output/file.out . index in any of the configured
local directories
2012-06-21 10:50:45,592 WARN org.apache.hadoop.mapred.TaskTracker:
getMapOutput (attempt_201206191809_0004_m_000006_0, 0) failed:
org.apache.hadoop.util.DiskChecker $ DiskErrorException: Could not find
taskTracker / ha.doop /
jobcache/job_201206191809_0004/attempt_201206191809_0004_m_0000
06_0/output/file.out.index in any of the configured local directories
Although two warn, but also affect the operating efficiency, they still try to resolve the cause of the error is unable to find a job in the middle of the output file. Need to make the following checks:
1. Configuration of mapred.local.dir property.
2. Df -h to see space in the cache path adequacy.
3. Free look at the memory space adequacy.
4. To ensure that the cache path writable permissions.
5. Check disk corruption.
The namenode cycle error is as follows:
2012-08-21 09:20:24,486 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit
log, edits.new files already exists in all healthy directories:
/ Data / work / hdfs / name / current / edits.new
/ Backup / current / edits.new
2012-08-21 09:20:25,357 ERROR
org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as: hadoop cause: java.net.ConnectException:
Connection refused
2012-08-21 09:20:25,357 ERROR
org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as: hadoop cause: java.net.ConnectException:
Connection refused
2012-08-21 09:20:25,359 WARN org.mortbay.log: / getimage:
java.io.IOException: GetImage failed. Java.net.ConnectException:
Connection refused
Related errors in secondarynamenode.
Search into an argument because:
With 1.0.2, only one checkpoint process is executed at a time. When the namenode gets an overlapping checkpointing request, it checks edit.new in its storage directories. If namenode have this file, namenode concludes the previous checkpoint process is not done yet and prints the warning message you've seen. This is the case if you ensure edits.new file before the error operation residual useless files can be deleted after the detection of whether there is such a problem.
Also make sure that the following namenode hdfs-site.xml configuration:
< Property>
< name>dfs.secondary.http.address </ name>
< value> 0.0.0.0:50090 </ value>
</ Property>
Above 0.0.0.0 modify your deployment secondarynamenode host name
secondarynamenode hdfs-site.xml the following items:
< Property>< name> dfs.http.address </ name>
< value> 0.0.0.0:50070 </ value>
</ Property>
0.0.0.0 modify the above namenode host name for your deployment
1, hadoop-root-datanode-master.log the following error:
ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
Incompatible namespaceIDs in
Lead to the datanode not start.
Cause: each namenode format will be re-created is a namenodeId and directory as dfs.data.dir configuration parameters contained in the directory configured by last format created by id and dfs.name.dir parameters id inconsistent.
cleared the data of namenode under the, namenode format but not emptied the datanode under the data, resulting in startup failure have to do is each time before format, the Empty dfs.data.dir parameter configuration directory.
Formatted HDFS command
Shell code
1. hadoop namenode-format
2, if the datanode not connect Namenode, resulting in datanode can not start.
ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
Call to ... failed on local exception: java.net.NoRouteToHostException: No route to
host
Turn off the firewall
Shell code
1. service iptables stop
The machine is rebooted, the firewall will open.
3, from the local to the HDFS file system to upload files, the following error:
INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException:
Bad connect ack with firstBadLink
INFO hdfs.DFSClient: Abandoning block blk_-1300529705803292651_37023
WARN hdfs.DFSClient: DataStreamer Exception: java.io.IOException: Unable to
create new block.
Solution:
Turn off the firewall:
Shell code
1. service iptables stop
Disable SELinux:
Edit / etc / selinux / config file, set "SELINUX = disabled"
4, safe mode errors caused by
org.apache.hadoop.dfs.SafeModeException: Cannot delete ..., Name node is in safe mode
Start in the distributed file system, the beginning of time there will be safe mode, when the case of the distributed file system in safe mode, the contents of the file system not allowed to modify can not be deleted until the end of the safe mode.
Safe Mode to check the of each DataNode data on the block effectiveness of the system starts, according to certain policies necessary to copy or delete the part of the data block. The runtime command can also be into safe mode. In practice, the system starts to modify, and delete files safe mode not allowed to modify the
error message will just have to wait a while you can.
Java code
1. hadoop dfsadmin-safemode leave
Turn off safe mode
Hadoop Common Errors with Possible Solution
Here I’m writing some of the Hadoop Issue faced by me and providing the solution with it hope you all get the benefit from it.
Hadoop cluster namenode formatted (bin / hadoop namenode -format)
restart cluster will appear as follows
1. After formatted name node (bin/hadoop namenode - format), Its come up with namespace Error,
ERROR: Incompatible namespaceIDS in ...: namenode namespaceID = ...,
datanode namespaceID = ...
Error because the format namenode will re-create a new namespaceID, so
that the original and datanode inconsistent.
Solution:
1. Data files deleted the datanode dfs.data.dir directory (default is tmp /
dfs / data)
2. Modify dfs.data.dir / current / VERSION file the namespaceID and
namenode identical to (log errors where there will be prompt)
3. To reassign new dfs.data.dir directory
2. Hadoop cluster is started with start-all.sh, slave always fail to start datanode, and will get an error:
ERROR: Could only be replicated to 0 nodes, instead of 1
Is the node identification may be repeated (personally think the wrong reasons). There may also be other reasons, and what solution then tries to solve.
Solution:
1. If port access, you should make sure the port is open, such as hdfs :/
/ machine1: 9000 / 50030,50070 like. Executive # iptables-I INPUT-p tcp-dport 9000-j ACCEPT command. If there is an error:
hdfs.DFSClient: Exception in createBlockOutputStream
java.net.ConnectException: Connection refused in; datanode port can not access, modify iptables: # iptables-I INPUT-s machine1-p tcp-j datanode on ACCEPT
2. There may be firewall restrictions between clusters to communicate with each other. Try to turn off the firewall. / Etc / init.d / iptables stop
3. Finally, there may be not enough disk space, check df -al
3. The program execution
Error: java.lang.NullPointerException
Null pointer exception, to ensure that the correct java program. Instantiated before the use of the variable what statement do not like array out of bounds. Inspection procedures.
When the implementation of the program, (various) error, make sure that the situation:
1. Premise of your program is correct by compiled
2. Cluster mode, the data to be processed wrote HDFS path and ensure correct
3. Specify the execution of jar package the entrance class name (I do not know why sometimes you do not specify also can run)
The correct wording similar to this:
$ hadoop jar myCount.jar myCount input output
4. Hadoop start datanode
ERROR: Unrecognized option:-jvm Could not the create the Java virtual machine.
Hadoop installation directory / bin / hadoop following piece of shell:
CLASS = 'org.apache.hadoop.hdfs.server.datanode.DataNode'
if [[$ EUID-eq 0]]; then
HADOOP_OPTS = "$ HADOOP_OPTS-jvm server $
HADOOP_DATANODE_OPTS"
else
HADOOP_OPTS = "$ HADOOP_OPTS-server $
HADOOP_DATANODE_OPTS"
if
$ EUID user ID, if it is the root of this identification will be 0, so try not to
use the root user to operate hadoop .
5.Terminal error message:
ERROR hdfs.DFSClient: Exception closing file / user / hadoop /
musicdata.txt: java.io.IOException: All datanodes 10.210.70.82:50010 are bad. Aborting ...
There are the jobtracker logs the error information
Error register getProtocolVersion
java.lang.IllegalArgumentException: Duplicate metricsName:
getProtocolVersion
And possible warning information:
WARN hdfs.DFSClient: DataStreamer Exception: java.io.IOException: Broken pipe
WARN hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_3136320110992216802_1063java.io.IOException: Connection reset by peer
WARN hdfs.DFSClient: Error Recovery for block
blk_3136320110992216802_1063 bad datanode [0] 10.210.70.82:50010
put: All datanodes 10.210.70.82:50010 are bad. Aborting ...
The solution:
1. Path of under the dfs.data.dir properties of whether the disk is full, try hadoop fs -put data if the processing is full again.
2. Related disk is not full, you need to troubleshoot related disk has no bad sectors, need to be detected.
6.Hadoop jar program get the error message:
java.io.IOException: Type mismatch in key from map: expected
org.apache.hadoop.io.NullWritable, recieved
org.apache.hadoop.io.LongWritable
Or something like this:
Status: FAILED java.lang.ClassCastException:
org.apache.hadoop.io.LongWritable cannot be cast to
org.apache.hadoop.io.Text
Then you need to learn the basics of Hadoop and map reduce model. In
"hadoop Definitive Guide book” in Chapter Hadoop I / O and in Chapter VII,
MapReduce type and format. If you are eager to solve this problem, I can also tell you a quick solution, but this is bound to affect you later development:
Ensure consistent data:
... Extends Mapper ...
public void map (k1 k, v1 v, OutputCollector output) ...
...
... Extends Reducer ...
public void reduce (k2 k, v2 v, OutputCollector output) ...
...
job.setMapOutputKeyClass (k2.class);
job.setMapOutputValueClass (k2.class);
job.setOutputKeyClass (k3.class);
job.setOutputValueClass (v3.class);
...
Note that the corresponding k * and v *. Recommendations or two chapters
I just said. Know the details of its principles.
7. If you hit a datanode error as follows:
ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
java.io.IOException: Cannot lock storage / data1/hadoop_data. The directory is already locked.
According to the error prompts view, it is the directory locked, unable to read. At this time you need to look at whether there are related process is still running or slave machine hadoop process is still running, use the linux
command to view:
Netstat -nap
ps-aux | grep Related PID
If hadoop related process is still running, use the kill command to kill can.
And then re-use start-all.sh.
8. If you encounter the jobtracker error follows:
ERROR: Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailingout.
Solution, modify datanode node /etc/hosts file.
Hosts under brief format:
Each line is divided into three parts: the first part of the network IP address, the second part of the host name or domain name, the third part of the host alias detailed steps are as follows:
1.first check the host name:
$ echo –e “ `hostname - i ` \t `hostname -n` \t $stn ”
Stn= short name or alies of hostname.
It will result in something like that
10.200.187.77 hadoop-datanode DN
If the IP address is configured on successfully modified, or show host name there is a problem, continue to modify the hosts file,
The shuffle error still appears this problem, then try to modify the configuration file of another user said hdfs-site.xml file, add the following:
dfs.http.address
*. *. *: 50070 The ports do not change, instead of the asterisk IP hadoop information transfer through HTTP, the port is same.
9.If you encounter the jobtracker error follows:
ERROR: java.lang.RuntimeException: PipeMapRed.waitOutputThreads ():
subprocess failed with code *
This is a java throws the system returns an error code, the meaning of the error code indicates details.
Please Excuse my typos and please share comment if you feel anything i left out.
10. If you encounter the following error:
FAILED java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI: ***
URI inside the characters not allowed, such as the colon: the class, the operating system does not allow the file named characters. In detail according to the prompt part (part asterisk) , Elimination of illegal character
can solve this issue.
11. To encounter tasktracker not start, tasktracker log error as follows:
ERROR org.apache.hadoop.mapred.TaskTracker: Cannot start task tracker because java.net.BindException: Address already in use ***
Ports are occupied or have corresponding process starts when you first stop the cluster, and then use ps-aux | grep hadoop command to look at the related process of hadoop , And Kill hadoop daemon .
12. Encounter datanode not started, the datanode log error is as follows:
ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
java.io.IOException: No locks available
No locks available can mean that you are trying to use hadoop on a filesystem that does not support file level locking.\
13. Are you trying to run your name node storage in NFS space?
Mentioned file-level locking, use
$ / Etc / init.d / nfs status
Command to view network file system, are closed. Another command df-Th
or mount the type of the file system, you can view the results obtained is indeed the NFS file system. Hanging in the network file system can not be used, because seemingly read-only, if not read-only situation, like said
above, does not support the file-level locking.
Finally the solution, you can try to add a file-level locking to nfs. I am here is to modify the dfs.data.dir, not to use nfs bin.
You can also try to format you Hadoop cluster (if it’s new one) and start all over again.
14. The datanode died, and you cannot start the process, log reported the following errors:
2012-06-04 10:31:34,915 INFO
org.apache.hadoop.hdfs.server.common.Storage: Cannot access storage
directory / data5/hadoop_data
2012-06-04 10:31:34,915 INFO
org.apache.hadoop.hdfs.server.common.Storage: Storage directory /data5/hadoop_data does not exist.
2012-06-04 10:31:35,033 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: org.apache.hadoop.util.DiskChecker $ DiskErrorException: Invalid value for volsFailed: 2, Volumes tolerated: 0
This problem, I found reason the nodes of the disk becomes read-only (only read) mode, online search, I found that quite a number of this case, Linux machine's hard disk could have been set to read-write (Read / Write) mode, but occasionally found automatically becomes read-only (Read Only), check some information, this may happens for various reasons, may be the problem:
· File system errors
· Kernel hardware driver bug
· The FW firmware classes problem
· Disk bad sectors
· Hard disk backplane fault
· Hard drive cable fault
· HBA card failure
· RAID card failure
· inode resource depletion
The solution:
· Restart the server (command reboot)
· Re-mount the hard disk
· fsck try to repair
· Replace the hard disk
15. When running mapreduce task error is as follows:
2012-06-21 10:50:43,290 WARN org.mortbay.log: / mapOutput:
org.apache.hadoop.util.DiskChecker $ DiskErrorException: Could not find
taskTracker/hadoop/jobcache/job_201206191809_0004/attempt_20120619
1809_0004_m_000006_0/output/file.out . index in any of the configured
local directories
2012-06-21 10:50:45,592 WARN org.apache.hadoop.mapred.TaskTracker:
getMapOutput (attempt_201206191809_0004_m_000006_0, 0) failed:
org.apache.hadoop.util.DiskChecker $ DiskErrorException: Could not find
taskTracker / ha.doop /
jobcache/job_201206191809_0004/attempt_201206191809_0004_m_0000
06_0/output/file.out.index in any of the configured local directories
Although two warn, but also affect the operating efficiency, they still try to resolve the cause of the error is unable to find a job in the middle of the output file. Need to make the following checks:
1. Configuration of mapred.local.dir property.
2. Df -h to see space in the cache path adequacy.
3. Free look at the memory space adequacy.
4. To ensure that the cache path writable permissions.
5. Check disk corruption.
The namenode cycle error is as follows:
2012-08-21 09:20:24,486 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit
log, edits.new files already exists in all healthy directories:
/ Data / work / hdfs / name / current / edits.new
/ Backup / current / edits.new
2012-08-21 09:20:25,357 ERROR
org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as: hadoop cause: java.net.ConnectException:
Connection refused
2012-08-21 09:20:25,357 ERROR
org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as: hadoop cause: java.net.ConnectException:
Connection refused
2012-08-21 09:20:25,359 WARN org.mortbay.log: / getimage:
java.io.IOException: GetImage failed. Java.net.ConnectException:
Connection refused
Related errors in secondarynamenode.
Search into an argument because:
With 1.0.2, only one checkpoint process is executed at a time. When the namenode gets an overlapping checkpointing request, it checks edit.new in its storage directories. If namenode have this file, namenode concludes the previous checkpoint process is not done yet and prints the warning message you've seen. This is the case if you ensure edits.new file before the error operation residual useless files can be deleted after the detection of whether there is such a problem.
Also make sure that the following namenode hdfs-site.xml configuration:
< Property>
< name>dfs.secondary.http.address </ name>
< value> 0.0.0.0:50090 </ value>
</ Property>
Above 0.0.0.0 modify your deployment secondarynamenode host name
secondarynamenode hdfs-site.xml the following items:
< Property>< name> dfs.http.address </ name>
< value> 0.0.0.0:50070 </ value>
</ Property>
0.0.0.0 modify the above namenode host name for your deployment
1, hadoop-root-datanode-master.log the following error:
ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
Incompatible namespaceIDs in
Lead to the datanode not start.
Cause: each namenode format will be re-created is a namenodeId and directory as dfs.data.dir configuration parameters contained in the directory configured by last format created by id and dfs.name.dir parameters id inconsistent.
cleared the data of namenode under the, namenode format but not emptied the datanode under the data, resulting in startup failure have to do is each time before format, the Empty dfs.data.dir parameter configuration directory.
Formatted HDFS command
Shell code
1. hadoop namenode-format
2, if the datanode not connect Namenode, resulting in datanode can not start.
ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
Call to ... failed on local exception: java.net.NoRouteToHostException: No route to
host
Turn off the firewall
Shell code
1. service iptables stop
The machine is rebooted, the firewall will open.
3, from the local to the HDFS file system to upload files, the following error:
INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException:
Bad connect ack with firstBadLink
INFO hdfs.DFSClient: Abandoning block blk_-1300529705803292651_37023
WARN hdfs.DFSClient: DataStreamer Exception: java.io.IOException: Unable to
create new block.
Solution:
Turn off the firewall:
Shell code
1. service iptables stop
Disable SELinux:
Edit / etc / selinux / config file, set "SELINUX = disabled"
4, safe mode errors caused by
org.apache.hadoop.dfs.SafeModeException: Cannot delete ..., Name node is in safe mode
Start in the distributed file system, the beginning of time there will be safe mode, when the case of the distributed file system in safe mode, the contents of the file system not allowed to modify can not be deleted until the end of the safe mode.
Safe Mode to check the of each DataNode data on the block effectiveness of the system starts, according to certain policies necessary to copy or delete the part of the data block. The runtime command can also be into safe mode. In practice, the system starts to modify, and delete files safe mode not allowed to modify the
error message will just have to wait a while you can.
Java code
1. hadoop dfsadmin-safemode leave
Turn off safe mode
No comments:
Post a Comment