Merge pull request #39237 from MikeSpreitzer/more-apf-obsservability

Extend APF Observability with logging and response headers

content/en/docs/concepts/cluster-administration/flow-control.md

@@ -638,6 +638,10 @@ poorly-behaved workloads that may be harming system health.
   standard deviation of seat demand seen during the last concurrency
   borrowing adjustment period.
 
+* `apiserver_flowcontrol_demand_seats_smoothed` is a gauge vector
+  holding, for each priority level, the smoothed enveloped seat demand
+  determined at the last concurrency adjustment.
+
 * `apiserver_flowcontrol_target_seats` is a gauge vector holding, for
   each priority level, the concurrency target going into the borrowing
   allocation problem.
@@ -761,7 +765,34 @@ serves the following additional paths at its HTTP[S] ports.
   system,            system-nodes,   12,         0,                   system:node:127.0.0.1, 2020-07-23T15:31:03.583823404Z, system:node:127.0.0.1, create, /api/v1/namespaces/scaletest/configmaps,
   system,            system-nodes,   12,         1,                   system:node:127.0.0.1, 2020-07-23T15:31:03.594555947Z, system:node:127.0.0.1, create, /api/v1/namespaces/scaletest/configmaps,
   ```
-  
+
+### Debug logging
+
+At `-v=3` or more verbose the server outputs an httplog line for every
+request, and it includes the following attributes.
+
+- `apf_fs`: the name of the flow schema to which the request was classified.
+- `apf_pl`: the name of the priority level for that flow schema.
+- `apf_iseats`: the number of seats determined for the initial
+  (normal) stage of execution of the request.
+- `apf_fseats`: the number of seats determined for the final stage of
+  execution (accounting for the associated WATCH notifications) of the
+  request.
+- `apf_additionalLatency`: the duration of the final stage of
+  execution of the request.
+
+At higher levels of verbosity there will be log lines exposing details
+of how APF handled the request, primarily for debug purposes.
+
+### Response headers
+
+APF adds the following two headers to each HTTP response message.
+
+- `X-Kubernetes-PF-FlowSchema-UID` holds the UID of the FlowSchema
+  object to which the corresponding request was classified.
+- `X-Kubernetes-PF-PriorityLevel-UID` holds the UID of the
+  PriorityLevelConfiguration object associated with that FlowSchema.
+
 ## {{% heading "whatsnext" %}}