Just want to share the experience when the min.user.id has not changed by default, no yarn application could be submitted. Usually, you will have something like below.
Diagnostics: Application application_1500628462670_0096 initialization failed (exitCode=255) with output: main : command provided 0
main : run as user is hiuy
main : requested yarn user is hiuy
Requested user hiuy is not whitelisted and has id 501,which is below the minimum allowed 1000
Failing this attempt. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
start time: 1500632228942
final status: FAILED
tracking URL: http://ip-172-31-23-181.ap-
southeast-1.compute.internal: 8088/cluster/app/application_ 1500628462670_0096
I2017-07-21 06:17:10,181 Client:[ForkJoinPool-1-worker-
3] Deleted staging directory hdfs://ip-172-31-16-195.ap- southeast-1.compute.internal: 8020/user/hiuy/.sparkStaging/ application_1500628462670_0096
E2017-07-21 06:17:10,185 SparkContext:[ForkJoinPool-1-
worker-3] Error initializing SparkContext.
SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
cluster. YarnClientSchedulerBackend. waitForApplication( YarnClientSchedulerBackend. scala:85) ~[spark-yarn_2.11-2.1.1.jar:2. 1.1]
cluster. YarnClientSchedulerBackend. start( YarnClientSchedulerBackend. scala:62) ~[spark-yarn_2.11-2.1.1.jar:2. 1.1]
TaskSchedulerImpl.start( TaskSchedulerImpl.scala:156) ~[spark-core_2.11-2.1.1.jar:2. 1.1]
(SparkContext.scala:509) ~[spark-core_2.11-2.1.1.jar:2. 1.1]
.getOrCreate(SparkContext. scala:2320) [spark-core_2.11-2.1.1.jar:2. 1.1]
SparkSession$Builder$$anonfun$ 6.apply(SparkSession.scala: 868) [spark-sql_2.11-2.1.1.jar:2.1. 1]
In order to solve the problem, you could either bump up the UID for the yarn application user to any integer value that above 1000, or you could set the min.user.id at YARN configuration. Make sure that you need to restart YARN when after the configuration alteration is done.
Hope that helps. Thanks!