Slave API 404s can fill up log files

[josephharrington]

We've created a dashboard for viewing console output data and if for any reason the console output is not available, the slave log file fills up with tons of errors. CR probably shouldn't log anything on a 404. We could probably also remove the line about the Access-Control-Allow-Origin header.

Example (hundreds of copies of this):

[2017-02-02 12:49:54.280] 19480 WARNING SlaveTornadoThr tornado.access  404 GET /v1/build/141/subjob/590/atom/10/console?max_lines=50 (10.8.189.29) 1.08ms
[2017-02-02 12:49:54.281] 19480 DEBUG   SlaveTornadoThr cluster_base_ha Origin "null" did not match cors_allowed_origins_regex conf value of "^http://[^.]*\.((inside-box)|(pod\.box)|(dev\.box))\.net". Not setting Access-Control-Allow-Origin header.
[2017-02-02 12:49:54.282] 19480 ERROR   SlaveTornadoThr cluster_base_ha Exception occurred during request to /v1/build/141/subjob/590/atom/12/console?max_lines=50.
Traceback (most recent call last):
  File "/home/jenkins/.virtualenvs/clusterrunner/lib/python3.4/site-packages/tornado/web.py", line 1346, in _when_complete
  File "/home/jenkins/.virtualenvs/clusterrunner/lib/python3.4/site-packages/tornado/web.py", line 1367, in _execute_method
  File "/home/jenkins/ClusterRunnerBuild/app/web_framework/cluster_slave_application.py", line 156, in get
  File "/home/jenkins/ClusterRunnerBuild/app/common/cluster_service.py", line 49, in get_console_output
app.util.exceptions.ItemNotFoundError: Output file doesn't exist for build_id: 141 subjob_id: 590 atom_id: 12
[2017-02-02 12:49:54.282] 19480 WARNING SlaveTornadoThr tornado.access  404 GET /v1/build/141/subjob/590/atom/12/console?max_lines=50 (10.8.189.29) 1.10ms
[2017-02-02 12:49:54.283] 19480 DEBUG   SlaveTornadoThr cluster_base_ha Origin "null" did not match cors_allowed_origins_regex conf value of "^http://[^.]*\.((inside-box)|(pod\.box)|(dev\.box))\.net". Not setting Access-Control-Allow-Origin header.
[2017-02-02 12:49:54.283] 19480 ERROR   SlaveTornadoThr cluster_base_ha Exception occurred during request to /v1/build/141/subjob/590/atom/3/console?max_lines=50.
Traceback (most recent call last):
  File "/home/jenkins/.virtualenvs/clusterrunner/lib/python3.4/site-packages/tornado/web.py", line 1346, in _when_complete
  File "/home/jenkins/.virtualenvs/clusterrunner/lib/python3.4/site-packages/tornado/web.py", line 1367, in _execute_method
  File "/home/jenkins/ClusterRunnerBuild/app/web_framework/cluster_slave_application.py", line 156, in get
  File "/home/jenkins/ClusterRunnerBuild/app/common/cluster_service.py", line 49, in get_console_output
app.util.exceptions.ItemNotFoundError: Output file doesn't exist for build_id: 141 subjob_id: 590 atom_id: 3

[tjlee0909]

I'm wondering if we really shouldn't log anything on a 404.

For sure, whether or not we do anything on the server side, the client side retry logic should stop retrying after a number of tries:

https://git.dev.box.net/Productivity/streaming-console/blob/master/src/console_control_base.jsx