Sqoop 1 job deployment: hcat credential error

Running the latest HUE 4.6 version. We’re trying to deploy a simple Oozie job involving Sqoop 1. However, if “hcat” is marked on Sqoop properties, i.e.,

when trying to run the job, the following error occurs:


[13/Feb/2020 16:48:12 +0200] exceptions_renderable ERROR    Potential detail: u'hcat'
[13/Feb/2020 16:48:12 +0200] exceptions_renderable ERROR    Potential trace: [('/usr/local/hue/apps/oozie/src/oozie/views/editor2.py', 415, '_submit_workf
low_helper', 'job_id = _submit_workflow(request.user, request.fs, request.jt, workflow, mapping)'), ('/usr/local/hue/apps/oozie/src/oozie/views/editor2.py
', 457, '_submit_workflow', 'job_id = submission.run()'), ('/usr/local/hue/desktop/libs/liboozie/src/liboozie/submission2.py', 59, 'decorate', 'deployment
_dir = self.deploy()'), ('/usr/local/hue/desktop/libs/liboozie/src/liboozie/submission2.py', 386, 'deploy', 'oozie_xml = self.job.to_xml(self.properties)'
), ('/usr/local/hue/apps/oozie/src/oozie/models2.py', 481, 'to_xml', "'workflow_mapping': workflow_mapping"), ('/usr/local/hue/desktop/core/src/desktop/lib/django_mako.py', 116, 'render_to_string_normal', 'result = template.render(**data_dict)'), ('/usr/local/hue/build/env/lib/python2.7/site-packages/Mako-1.0.7-py2.7.egg/mako/template.py', 462, 'render', 'return runtime._render(self, self.callable_, args, data)'), ('/usr/local/hue/build/env/lib/python2.7/site-packages/Mako-1.0.7-py2.7.egg/mako/runtime.py', 838, '_render', '**_kwargs_for_callable(callable_, data))'), ('/usr/local/hue/build/env/lib/python2.7/site-packages/Mako-1.0.7-py2.7.egg/mako/runtime.py', 873, '_render_context', '_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)'), ('/usr/local/hue/build/env/lib/python2.7/site-packages/Mako-1.0.7-py2.7.egg/mako/runtime.py', 899, '_exec_template', 'callable_(context, *args, **kwargs)'), ('/tmp/tmpStZ7YH/oozie/editor2/gen/workflow.xml.mako.py', 66, 'render_body', "credential = mapping['credentials'][cred_type]")]
[13/Feb/2020 16:48:12 +0200] middleware   INFO     Processing exception: Workflow submission failed: Traceback (most recent call last):
  File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/utils/decorators.py", line 185, in inner
    return func(*args, **kwargs)
  File "/usr/local/hue/apps/oozie/src/oozie/decorators.py", line 115, in decorate
    return view_func(request, *args, **kwargs)
  File "/usr/local/hue/apps/oozie/src/oozie/decorators.py", line 77, in decorate
    return view_func(request, *args, **kwargs)
  File "/usr/local/hue/apps/oozie/src/oozie/views/editor2.py", line 374, in submit_workflow
    return _submit_workflow_helper(request, workflow, submit_action=reverse('oozie:editor_submit_workflow', kwargs={'doc_id': workflow.id}))
  File "/usr/local/hue/apps/oozie/src/oozie/views/editor2.py", line 417, in _submit_workflow_helper
    raise PopupException(_('Workflow submission failed'), detail=smart_str(e), error_code=200)
PopupException: Workflow submission failed

What configuration might be missing here? The error is not clear, what is Hue trying to access that is failing, and which setting could fix that?

I assume that some setting may be missing since we migrated from HDP 2.6 to 3.1, and HUE worked without problems with older Hive/Hcat; but nothing obvious comes to mind, the hive core.xml and similar does not seem to differ much, plus both Hcat ant Sqoop runs without problems if created manuually. The problem is only when trying to create the jobs using HUE.

Will appreciate any ideas!

Would recommend first to upgrade to Hue 4.6 which is much more robust.

Then there is probably something missing in the fetching of the credentials or a bug:


Thanks, we are running Hue 4.6.0 and, except for this issue, it’s great! I’ll try to dive deeper into the code. My guess is that some config gets lost during the creation of hive_properties object, however not yet sure how to properly debug this.

Romain, thanks again for showing the right direction, I’ve managed to solve it. It was actually a misconfiguration on oozie-site.xml, oozie.credentials.credentialclasses for some reason was configured to use hcat=org.apache.oozie.action.hadoop.HiveCredentials instead of hcat=org.apache.oozie.action.hadoop.HCatCredentials. Switching this back resolved the issue from HUE.

Since this was set as default with HDP 3.1, it’s hard to tell yet whether this was intentional or not, so will have to see whether further issues will come up.

1 Like