Fetching working logs for oozie

#1

Where is HUE taking Oozie logs from during every step of a workflow?

On a normal flow, I’m used to seeing logs and a progress bar, e.g.,:
image

However, on a different cluster, where HUE is set up in the same manner, there is no progress and logs at all, the icon is empty during execution (in Running state):
image

Meanwhile there are no errors reported in HUE client. The user has access to the logs in yarn, and they can be accessed in yarn resourcemanager, but for some reason are not shown in HUE.

This seems like a permission issue, but I’m not sure where to start the debug.

#2

IIRC those come from the YARN jobs spawn by the action (better to check the Oozie launcher logs if there or failure too).

#3

Thanks. But I’ve noticed that we can see the logs and the errors when clicking on “Tasks”, e.g.,

image
image839×190 10.1 KB

However, at the same time, the “Graph” view doesn’t show the progress and which task is running at the moment (or which caused the error).

On the other hand, these jobs seem to “expire” very quickly. Often, after an finishing the task, clicking through it returns, e.g.,
image

Maybe it uses a different port or channel for getting the logs from yarn/oozie? The “Graph” view was very convenient, and I’m just trying to debug which part is breaking here.

#4

@Romain

In /var/log/hue/error.log I’m noticing these error related to the jobbrowser:

[06/Dec/2020 06:43:31 +0200] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 500 response
[06/Dec/2020 06:43:32 +0200] api2         ERROR    Error running <function logs at 0x7f6cc3ec1b18>
Traceback (most recent call last):
  File "/usr/local/hue/apps/jobbrowser/src/jobbrowser/api2.py", line 40, in decorator
    return func(*args, **kwargs)
  File "/usr/local/hue/apps/jobbrowser/src/jobbrowser/api2.py", line 120, in logs
    response['logs'] = get_api(request.user, interface, cluster=cluster).logs(app_id, app_type, log_name, json.loads(request.GET.get('is_embeddable', 'false')
.lower()))
  File "/usr/local/hue/apps/jobbrowser/src/jobbrowser/apis/job_api.py", line 62, in logs
    return self._get_api(appid).logs(appid, app_type, log_name, is_embeddable)
  File "/usr/local/hue/apps/jobbrowser/src/jobbrowser/apis/job_api.py", line 231, in logs
    response = job_single_logs(MockDjangoRequest(self.user), job=appid)
  File "/usr/local/hue/apps/jobbrowser/src/jobbrowser/views.py", line 90, in decorate
    return view_func(request, *args, **kwargs)
  File "/usr/local/hue/apps/jobbrowser/src/jobbrowser/views.py", line 427, in job_single_logs
    failed_tasks = job.filter_tasks(task_states=('failed',))
  File "/usr/local/hue/apps/jobbrowser/src/jobbrowser/yarn_models.py", line 330, in filter_tasks
    tasks = self.api.tasks(self.id).get('tasks', {})
  File "/usr/local/hue/desktop/libs/hadoop/src/hadoop/yarn/mapreduce_api.py", line 118, in tasks
    return self._root.get('%(app_id)s/ws/%(version)s/mapreduce/jobs/%(job_id)s/tasks' % {'app_id': app_id, 'job_id': job_id, 'version': _API_VERSION}, params=self._get_params(), headers={'Accept': _JSON_CONTENT_TYPE})
  File "/usr/local/hue/desktop/core/src/desktop/lib/rest/resource.py", line 157, in get
    return self.invoke("GET", relpath, params, headers=headers, allow_redirects=True, clear_cookies=clear_cookies)
  File "/usr/local/hue/desktop/core/src/desktop/lib/rest/resource.py", line 87, in invoke
    log_response=log_response
  File "/usr/local/hue/desktop/core/src/desktop/lib/rest/resource.py", line 114, in _invoke
    clear_cookies=clear_cookies
  File "/usr/local/hue/desktop/core/src/desktop/lib/rest/http_client.py", line 229, in execute
    raise self._exc_class(ex)
RestException: 500 Server Error: Server Error for url: http://fqdn:8088/proxy/application_1607229664670_0001/ws/v1/mapreduce/jobs/job_1607229664670_0001/tasks?doAs=etl

However, I’m not able to reproduce the error through curl and hue user. Is this still an issue with the yarn configuration, or could there be some issues with python packages used for the http calls?