Sqoop 1 job deployment: hcat credential error

operator · February 13, 2020, 3:00pm

Running the latest HUE 4.6 version. We’re trying to deploy a simple Oozie job involving Sqoop 1. However, if “hcat” is marked on Sqoop properties, i.e.,

when trying to run the job, the following error occurs:

runcpserver.log:

[13/Feb/2020 16:48:12 +0200] exceptions_renderable ERROR    Potential detail: u'hcat'
[13/Feb/2020 16:48:12 +0200] exceptions_renderable ERROR    Potential trace: [('/usr/local/hue/apps/oozie/src/oozie/views/editor2.py', 415, '_submit_workf
low_helper', 'job_id = _submit_workflow(request.user, request.fs, request.jt, workflow, mapping)'), ('/usr/local/hue/apps/oozie/src/oozie/views/editor2.py
', 457, '_submit_workflow', 'job_id = submission.run()'), ('/usr/local/hue/desktop/libs/liboozie/src/liboozie/submission2.py', 59, 'decorate', 'deployment
_dir = self.deploy()'), ('/usr/local/hue/desktop/libs/liboozie/src/liboozie/submission2.py', 386, 'deploy', 'oozie_xml = self.job.to_xml(self.properties)'
), ('/usr/local/hue/apps/oozie/src/oozie/models2.py', 481, 'to_xml', "'workflow_mapping': workflow_mapping"), ('/usr/local/hue/desktop/core/src/desktop/lib/django_mako.py', 116, 'render_to_string_normal', 'result = template.render(**data_dict)'), ('/usr/local/hue/build/env/lib/python2.7/site-packages/Mako-1.0.7-py2.7.egg/mako/template.py', 462, 'render', 'return runtime._render(self, self.callable_, args, data)'), ('/usr/local/hue/build/env/lib/python2.7/site-packages/Mako-1.0.7-py2.7.egg/mako/runtime.py', 838, '_render', '**_kwargs_for_callable(callable_, data))'), ('/usr/local/hue/build/env/lib/python2.7/site-packages/Mako-1.0.7-py2.7.egg/mako/runtime.py', 873, '_render_context', '_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)'), ('/usr/local/hue/build/env/lib/python2.7/site-packages/Mako-1.0.7-py2.7.egg/mako/runtime.py', 899, '_exec_template', 'callable_(context, *args, **kwargs)'), ('/tmp/tmpStZ7YH/oozie/editor2/gen/workflow.xml.mako.py', 66, 'render_body', "credential = mapping['credentials'][cred_type]")]
[13/Feb/2020 16:48:12 +0200] middleware   INFO     Processing exception: Workflow submission failed: Traceback (most recent call last):
  File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/hue/build/env/lib/python2.7/site-packages/Django-1.11.22-py2.7.egg/django/utils/decorators.py", line 185, in inner
    return func(*args, **kwargs)
  File "/usr/local/hue/apps/oozie/src/oozie/decorators.py", line 115, in decorate
    return view_func(request, *args, **kwargs)
  File "/usr/local/hue/apps/oozie/src/oozie/decorators.py", line 77, in decorate
    return view_func(request, *args, **kwargs)
  File "/usr/local/hue/apps/oozie/src/oozie/views/editor2.py", line 374, in submit_workflow
    return _submit_workflow_helper(request, workflow, submit_action=reverse('oozie:editor_submit_workflow', kwargs={'doc_id': workflow.id}))
  File "/usr/local/hue/apps/oozie/src/oozie/views/editor2.py", line 417, in _submit_workflow_helper
    raise PopupException(_('Workflow submission failed'), detail=smart_str(e), error_code=200)
PopupException: Workflow submission failed

What configuration might be missing here? The error is not clear, what is Hue trying to access that is failing, and which setting could fix that?

I assume that some setting may be missing since we migrated from HDP 2.6 to 3.1, and HUE worked without problems with older Hive/Hcat; but nothing obvious comes to mind, the hive core.xml and similar does not seem to differ much, plus both Hcat ant Sqoop runs without problems if created manuually. The problem is only when trying to create the jobs using HUE.

Will appreciate any ideas!

Romain · February 13, 2020, 4:40pm

Would recommend first to upgrade to Hue 4.6 which is much more robust.

Then there is probably something missing in the fetching of the credentials or a bug:

github.com

cloudera/hue/blob/master/desktop/libs/liboozie/src/liboozie/credentials.py

#!/usr/bin/env python
# Licensed to Cloudera, Inc. under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  Cloudera, Inc. licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from builtins import object
import logging

This file has been truncated. show original

credentials.fetch(api)

github.com

cloudera/hue/blob/master/apps/oozie/src/oozie/views/editor2.py#L110


  return _edit_workflow(request, doc, workflow)




def _edit_workflow(request, doc, workflow):
  workflow_data = workflow.get_data()


  api = get_oozie(request.user)
  credentials = Credentials()


  try:
    credentials.fetch(api)
  except Exception as e:
    LOG.error(smart_str(e))


  can_edit_json = doc is None or (doc.can_write(request.user) if USE_NEW_EDITOR.get() else doc.doc.get().is_editable(request.user))


  return render('editor2/workflow_editor.mako', request, {
      'layout_json': json.dumps(workflow_data['layout'], cls=JSONEncoderForHTML),
      'workflow_json': json.dumps(workflow_data['workflow'], cls=JSONEncoderForHTML),
      'credentials_json': json.dumps(list(credentials.credentials.keys()), cls=JSONEncoderForHTML),
      'workflow_properties_json': json.dumps(WORKFLOW_NODE_PROPERTIES, cls=JSONEncoderForHTML),

operator · February 13, 2020, 5:09pm

Thanks, we are running Hue 4.6.0 and, except for this issue, it’s great! I’ll try to dive deeper into the code. My guess is that some config gets lost during the creation of hive_properties object, however not yet sure how to properly debug this.

operator · February 13, 2020, 5:39pm

Romain, thanks again for showing the right direction, I’ve managed to solve it. It was actually a misconfiguration on oozie-site.xml, oozie.credentials.credentialclasses for some reason was configured to use hcat=org.apache.oozie.action.hadoop.HiveCredentials instead of hcat=org.apache.oozie.action.hadoop.HCatCredentials. Switching this back resolved the issue from HUE.

Since this was set as default with HDP 3.1, it’s hard to tell yet whether this was intentional or not, so will have to see whether further issues will come up.